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OVERVIEW 


The Behavioral Interventions to Advance Self-Sufficiency (BIAS) project is spon- 
sored by the Office of Planning, Research and Evaluation of the Administration for 
Children and Families (ACF) in the U.S. Department of Health and Human Services. 
The project, led by MDRC, aims to apply behavioral insights to issues related to the 
operations, implementation, and efficacy of selected programs and policies. 

This report describes a collaboration between the Indiana Office of Early Childhood 
and Out-of-School Learning (OECOSL) and the BIAS team. The OECOSL is the lead 
agency responsible for administering the state’s Child Care and Development Fund 
(CCDF), which provides child care subsidies to low-income parents who are work- 
ing or in school. The BIAS team tested three behavioral interventions related to the 
CCDF program using random assignment. 

The first intervention focused on child care decision making among low-income 
parents. It aimed to increase the percentage of parents who used their CCDF subsi- 
dies to pay for providers in the state’s quality rating and improvement system, called 
Paths to QUALITY (PTQ). The BIAS team replaced the letter and brochure typically 
sent to parents on the CCDF waitlist with a redesigned packet, which included indi- 
vidualized child care referrals. Some parents also received a personal phone call. 

The intervention ran for four months. Findings showed that it did not increase the 
overall percentage of CCDF families who chose any PTQ provider; however, being 
sent a referral list combined with a phone call increased the percentage of parents 
who chose a highly rated provider by 2.1 percentage points (a 17 percent change). 

Two additional tests focused on the CCDF redetermination process. Parents in Indi- 
ana had to verify their eligibility at least every six months. The interventions aimed 
to encourage parents to attend their first scheduled appointment and to help par- 
ents complete the process in one appointment. If successful, the intervention would 
reduce the hassles of redetermination, which may affect the continuity of care. The 
BIAS team replaced the notification materials normally sent to parents with a rede- 
signed appointment letter, checklist, and reminder postcard. 

The first test ran for five months in Marion County. Findings showed that the 
behavioral intervention increased the percentage of parents who attended their 
first scheduled redetermination appointment by 2.6 percentage points (a 5 percent 
change) and the percentage of parents who completed redetermination in one 
appointment by 3.2 percentage points (a 5 percent change). It did not change the 
likelihood that parents would renew on time. 

Using a rapid-cycle evaluation approach, the team designed a second intervention 
using lessons from the first. This test ran for four months. The second intervention 
increased the percentage of parents who attended their first scheduled redetermina- 
tion appointment by 10.6 percentage points (a 24 percent change). It did not change 
the likelihood that parents completed redetermination in one appointment, but it did 
increase the percentage of parents who renewed on time by 2.7 percentage points 
(a 4 percent change). 
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EXECUTIVE 

SUMMARY 


The Behavioral Interventions to Advance Self-Sufficiency (BIAS) project is the first major opportunity to use a 
behavioral economics lens to examine programs that serve poor and vulnerable families in the United States. 
Sponsored by the Office of Planning, Research and Evaluation (OPRE) of the Administration for Children and 
Families (ACF) in the U.S. Department of Health and Human Services and led by MDRC, the project applies 
behavioral insights to issues related to the operations, implementation, and efficacy of social service pro- 
grams and policies. The goal is to learn how tools from behavioral science can be used to deliver programs 
more effectively and, ultimately, improve the well-being of low-income children, adults, and families. 

This report describes a collaboration between the Indiana Office of Early Childhood and Out-of-School 
Learning (OECOSL) and the BIAS team. The OECOSL is the lead agency responsible for administering 
the state’s Child Care and Development Fund (CCDF), which provides child care subsidies to low-income 
parents who are working or in school. The CCDF has the dual goals of supporting parental employment and 
furthering children’s development. Over a period of 19 months, the BIAS team designed and evaluated three 
behavioral interventions that aimed to improve outcomes at two important points in the child care program 
— when parents enroll in the CCDF program and must select a child care provider, and when they renew 
their subsidies. 

The first intervention, which was launched statewide in June 2014 and ran through October 2014, 
focused on child care decision making among low-income parents. It aimed to increase the percentage of 
parents who used their CCDF subsidies to pay for highly rated providers in the state’s quality rating and 
improvement system — Paths to QUALITY (PTQ). The PTQ program ranks child care providers on a four- 
point scale based on standards related to health and safety, staff qualifications, parental engagement, 
and curriculum development. The OECOSL sought to increase the percentage of CCDF parents who 
selected PTQ providers, and to increase the number who chose highly rated providers (Levels 3 or 4). 

Two additional interventions, which ran from January through November 2014, focused on CCDF 
redetermination in Marion County, a large urban county that includes the city of Indianapolis. Parents 
had to verify their continued eligibility for child care subsidies at least every six months by submitting 
required documentation. The OECOSL aimed to use behavioral insights to encourage parents to attend 
their first scheduled appointment and to complete the process in one visit. 

BIAS Diagnosis and Design Process 

The BIAS team used a method called “behavioral diagnosis and design” to identify potential behavioral 
“bottlenecks” and test low-cost, behaviorally informed interventions to address them. The behavioral 
diagnosis and design process consists of four phases. 

• Define: The BIAS team defines the problem in a way that is precise enough to test. 

• Diagnose: The team collects both qualitative and quantitative data to identify factors (“bottle- 
necks”) that may be causing the problem, and uses the data to develop theories based on behavioral 

research about why bottlenecks are occurring. 
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• Design: The team uses theories about why bottlenecks are occurring and other behavioral 
insights to develop an intervention. 

• Test: The team evaluates the behavioral intervention using rigorous scientific methods. 

The process is ideally iterative, allowing for multiple rounds of hypothesis development and testing, 
and aims to connect the problem, behavioral bottleneck, and design solution together in a coherent way. 

The rest of this section describes how the BIAS team applied this process to Indiana’s child care 
programs. 

Use of Quality-Rated Care by Recipients of CCDF Subsidies 

Research has shown that high-quality early childhood and out-of-school-time experiences can have a posi- 
tive impact on the lives of children, particularly those from economically distressed households. 1 The CCDF 
program expands parents’ access to child care by subsidizing the cost, but, with an emphasis on parental 
choice, there is no requirement that parents select a high-quality provider. 

To address this issue, the OECOSL has tried to encourage CCDF parents to use providers that par- 
ticipate in the PTQ system. 

Define 

About 65 percent of CCDF families in Indiana were enrolled with PTQ providers as of March 2013. The 
OECOSL was interested in increasing the percentage of parents who selected a PTQ provider at the time 
of enrollment and the percentage of parents who selected the highest-rated providers (Levels 3 or 4). 

Diagnose 

Indiana traditionally has more demand for its CCDF subsidies than available funds, and administrators 
place families they cannot serve immediately on a waitlist. Before engaging with the BIAS team, the 
OECOSL marketed the PTQ program to parents on the waitlist by mailing them a letter and brochure 
about PTQ. The theory was that parents would use this time to research and select a PTQ provider. 
Parents on the waitlist might not have a stable form of child care, or might be using a provider that is not 
eligible to accept the CCDF subsidy. The subsidy provides parents with additional resources to pay for 
previously unaffordable child care options. 

The BIAS team investigated why some parents who enroll in CCDF do not choose quality-rated 
providers. The team analyzed OECOSL data on parents’ child care choices; conducted interviews with 
agency staff and clients to better understand their experiences; and reviewed a variety of forms, let- 
ters, and flowcharts relevant to the CCDF and PTQ programs. The diagnosis pointed to two factors that 
generally constrain parents’ child care choices that were outside the scope of the BIAS project, which 
focuses on relatively quick and inexpensive changes. The first is access to PTQ providers, especially 
those with high ratings. Only about a third of PTQ providers are highly rated (Level 3 or 4), and it is 
difficult to know how many of these have vacancies and impracticable to determine whether they are 
located in places that are convenient to CCDF parents. The second factor is cost. Many parents face out- 
of-pocket expenses for child care, even when the provider accepts a CCDF subsidy; the exact amount is 
determined by the provider and generally increases with the quality of care. 

As a result, the behavioral diagnosis focused on parents’ awareness of quality-rated providers, 
criteria for selecting child care, and opportunity to choose a quality-rated provider. The team identified 
bottlenecks related to selecting a child care provider and theorized behavioral reasons for the bottle- 
necks. 


1 James J. Heckman, “Skill Formation and the Economics of Investing in Disadvantaged Children,” Science 312, 5782: 1900-1902 
(2006); Lynn A. Karoly, “Investing in the Future: Reducing Poverty Through Human Capital Investments,” Focus 21, 2: 38-43 
(2000); Janet Currie, “Early Childhood Education Programs,” Journal of Economic Perspectives 15, 2: 213-238 (2001). 
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Design 

Based on the findings from the behavioral diagnosis, the BIAS team replaced the letter and brochure 
typically sent to parents with a redesigned packet that included individualized child care referrals. 

Some parents also received a phone call. The Indiana Association for Child Care Resource and Referral 
(IACCRR), the statewide child care resource and referral agency, had the existing capacity and resources 
to create the referrals and reach out to a subset of parents by phone. The BIAS packet contained the 
behavioral elements of personalization, simplification, implementation prompts, and visual cues. The 
materials included: 

• A redesigned letter, personalized with the parent's name and a note, both handwritten. The 
letter also contained an implementation prompt to guide the parent in the search for child 
care. 2 

• A graphic depicting the four levels of quality and explaining the benefits of choosing a PTQ 
provider and a simplified checklist outlining reasons to select a PTQ provider. 

• A customized map and three individualized child care referrals created using software that 
IACCRR developed for the BIAS intervention. The map presented a snapshot of three provid- 
ers near the parent’s home with key information about each provider, including PTQ ratings. 

The three newly designed, personalized referrals were for the same providers listed on the 
map and arranged from the highest PTQ rated to the unrated. 

• A checklist of questions to ask when visiting a potential child care provider. 

Test 

Intervention materials were sent from June 2014 to October 2014 and the BIAS team evaluated it using a 
random assignment design, whereby the team randomly divided the sample into three study groups: 

1 . Control group — was sent the standard OECOSL letter and brochure 

2. Referrals-only group — was sent the BIAS packet with referrals 

3. Referrals-phone group — was sent the BIAS packet with referrals and received a phone call from 
IACCRR 

In the first week, the BIAS team randomly assigned parents whose children were currently on the 
CCDF waitlist and forwarded the assignments to IACCRR. These families make up Cohort 1 and include 
5,949 children, many of whom had likely been on the waitlist for months. Beginning in the second week, 
the team randomly assigned parents on a rolling basis within about a week of them signing up for the 
waitlist. These families make up Cohort 2 and include 6,703 children. Thus, parents in Cohort 2 were 
sent materials close to when they signed up for the waitlist, whereas many parents in Cohort 1 received 
the materials after a substantial delay from when they signed up. The study found: 

• Less than half of the children were enrolled in CCDF during the 12 months of study and 
follow-up. There was no difference in the enrollment rate between study groups. 

• For both cohorts pooled, the BIAS interventions did not increase the percentage of families 
who selected PTQ providers. However, the referrals-phone intervention increased the use 
of highly rated providers (Level 3 or 4) by 2.1 percentage points. The increase was almost 
entirely due to a 2.0 percentage point, or 33 percent, increase in the use of Level 4 providers. 


2 Implementation prompt is a common term for a strategy designed to encourage individuals to make detailed plans to fulfill 
goals. Katherine L. Milkman, John Beshears, James Choi, David Laibson, and Brigitte Madrian, “Following Through on 
Good Intentions: The Power of Planning Prompts,” NBER Working Paper No. 17995, (Cambridge, MA: National Bureau of 
Economic Research, 2012); Katherine L. Milkman, John Beshears, James Choi, David Laibson, and Brigitte Madrian, “Using 
Implementation Intentions Prompts to Enhance Influenza Vaccination Rates,” Proceedings of the National Academy of Sciences 
108, 26: 10415-10420 (2012). 
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TABLE ES.l 

PROVIDER SELECTION OUTCOMES, PTQ TEST 

INDIANA OFFICE OF EARLY CHILDHOOD AND OUT-OF-SCHOOL LEARNING 


OUTCOME 

CONTROL 

GROUP 

REFERRALS- 

ONLY 

GROUP 

DIFFERENCE 

REFERRALS- 

PHONE 

GROUP 

DIFFERENCE 

Enrollment 

Enrolled in the CCDF program (%) 

42.0 

42.3 

0.2 

44.7 

2.6 

Provider quality rating selection 

Selected an unrated provider (%) 

13.4 

14.4 

1.0 

14.2 

0.8 

Selected a PTQ provider (%) 

28.7 

27.9 

-0.7 

30.4 

1.8 

Level 3 or 4 provider 

12.6 

13.2 

0.6 

14.7 

2.1* 

Level 4 provider 

6.0 

6.7 

0.8 

8.0 

2.0** 

Level 3 provider 

6.6 

6.4 

-0.2 

6.8 

0.2 

Level 2 provider 

2.9 

3.3 

0.4 

2.5 

-0.5 

Level 1 provider 

13.2 

11.5 

-1.7* 

13.3 

0.1 

Sample size (total = 12,652) 

6,657 

3,580 


2,415 



SOURCE: MDRC calculations based on data from the Office of Early Childhood and Out-of-School Learning. 

NOTES: Estimates were regression-adjusted using ordinary least squares, controlling for pre-random assignment characteristics 
of sample members. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Rounding may cause slight discrepancies in sums and differences. 


(See Table ES.l.) The referrals-only intervention decreased the percentage that chose a Level 1 
provider by 1.7 percentage points, a 13 percent change. 

• The effectiveness of the BIAS interventions differed by cohort. 3 In Cohort 2, the referrals- 
phone group was more likely to select a highly rated (Level 3 or 4) provider by 4 percentage 
points, or 31 percent. (See Figure ES.l.) The referrals-phone group in Cohort 2 was also more 
likely to choose a Level 4 provider by 2.7 percentage points, or 45 percent. 

Thus, the behavioral interventions tested in this study did not change the likelihood that parents 
would choose a PTQ provider, but they did shift preferences among those who chose a quality-rated 
provider to the higher end of the quality spectrum. This finding was only clearly seen for parents who 
received the phone call in addition to the individualized referrals in the full sample, although the sub- 
group analysis shows that the increase in choosing a Level 4 provider was also present among parents 
who received only referrals shortly after they signed up for the waitlist (the Cohort 2 referrals-only group). 
These effects are meaningful given the challenges many parents face finding suitable child care. 

Renewing CCDF Subsidies 

While the PTQ intervention focused on parents who were waiting to receive a CCDF voucher, the second 
set of interventions focused on parents who were already receiving a CCDF subsidy. In order to continue 
receiving a subsidy in Indiana, parents had to prove they were still eligible about every six months by 
providing documentation. While periodic redetermination is necessary to maintain the integrity of the 
program, parents often struggle to comply with this requirement. Difficulties during the redetermination 
process can threaten the continuity of care, which, in turn, can negatively affect children’s development, 
parents’ employment, and the efficient administration of the CCDF program. 


3 These effects were not statistically significantly different from the lack of effects in Cohort 1 and thus should be treated with 
caution. 
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FIGURE ES.l 

HIGHLY RATED PROVIDER SELECTION IN COHORT 2, PTQ TEST 

INDIANA OFFICE OF EARLY CHILDHOOD AND OUT-OF-SCHOOL LEARNING 


18.0 -r 


B 15.0 + 

Vi 
> 
o 

y-i 

a 

O 
E-* 
a 

o 


12.0 -- 


>? 

a 

a 

2 

0 

O 

0 

'0 

0 

O 

a 


0 

0 


0 

a 


9.0 - 


6.0 - 


3.0 -- 


1.3 



Control 


Referrals-only 


4.0*** 


16.8 


Referrals-phone 


SOURCE: MDRC calculations based on data from the Office of Early Childhood and Out-of-School Learning. 

NOTES: The numbers above the two right bars indicate the difference in selection rates between that group and the control group. 

Estimates were regression-adjusted using ordinary least squares, controlling for pre-random assignment characteristics 
of sample members. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Rounding may cause slight discrepancies in differences. 


Define 

The OECOSL wanted to use behavioral insights to increase the percentage of parents who attend their 
first scheduled appointment, complete the redetermination process in one appointment, and complete 
the process on time. Data reported by the OECOSL and its contractor Children’s Bureau, Inc. (CBI), sug- 
gested that about 40 percent of parents missed their scheduled appointment date, about a third who 
attended any appointment had to attend multiple appointments to complete the process, and about 17 
percent did not renew their subsidies. 

Diagnose 

During the diagnosis phase, the BIAS team collected data from the OECOSL and CBI about the redeter- 
mination process and spoke to frontline and supervisory staff at the intake office, the regional OECOSL 
manager of the intake offices, and some parents who had just attended their redetermination appoint- 
ments. The team identified two contextual factors that made redetermination challenging. First, par- 
ents must visit the intake office to complete redetermination, rather than sending forms electronically 
or by mail as some other benefits programs allow. The second factor relates to specific redetermination 
requirements, all of which are determined by policy or statute. Parents often find fulfilling some require- 
ments difficult, particularly those related to employment verification that require dated pay stubs or 
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other documents that may not cover the 30-day period before the appointment. Addressing these barri- 
ers was beyond the scope of the BIAS project. 

The BIAS team instead identified a number of bottlenecks related to renewing child care benefits. 
The team then launched two rounds of redetermination tests in Indiana, an example of rapid-cycle evalu- 
ation. The first test was launched in January 2014 and the second in June 2014, based on findings from 
Round 1. The design and the test of each round are discussed below. 

Design — Round 1 

In Round 1, the BIAS team redesigned the appointment materials and created a new reminder postcard. 
The redesigned materials incorporated the behavioral elements of simplification, personalization, loss 
aversion, and reminders. 

• The redesigned appointment letter excluded unnecessary content and presented only the es- 
sential information that parents needed to remember. It began with a handwritten greeting, 
which aimed to personalize the interaction, and prominently displayed the subsidy expiration 
date to elicit a sense of urgency. 4 

• A simplified checklist served as a plan-making device. It contained guiding questions to help 
parents decide which documents to bring within each of the four required categories (identifi- 
cation, service need and other forms of income, address, and provider information). 

• The provider information form was printed on yellow paper to make it more salient. 

• A reminder postcard, similar to those sent out for doctors’ appointments, was mailed about 
one to two weeks before the appointment. It was not personalized but included a box where 
parents could write in their appointment time. 

Test — Round 1 

In each round, the BIAS team evaluated the intervention using a random assignment design. It divided 
sample members equally between an intervention group and a control group. (Figure ES.2 displays the 
findings from both redetermination rounds.) 

The Round 1 test, in which the team sent intervention materials from January to June 2014, found that 
the BIAS materials: 

• Increased the percentage of parents who attended their first scheduled redetermination ap- 
pointment by 2.6 percentage points, a 5 percent change 

• Increased the percentage of parents who completed the redetermination process in one ap- 
pointment by 3.2 percentage points, a 5 percent change 5 

• Did not change the likelihood that parents would renew their subsidies on time 

Design — Round 2 

Given that the effects were not as large as desired, the BIAS team conducted a rapid-cycle evaluation and 
designed a second intervention based on findings from Round 1. In Round 2, the BIAS team retained the 
simplified checklist tested in Round 1, but otherwise revised the intervention materials in three critical 
ways: 

• The appointment letter was again redesigned to resemble a vertical roadmap, with the ap- 
pointment date and time as the midpoint. This design sought to counteract the possibility 


4 The BIAS team slightly modified this letter during the test period after receiving feedback from CCDF staff that some parents 
mistook the expiration date for the appointment date. 

5 Data on whether or not parents renewed after the deadline were not available, so this analysis reported on the number of 
appointments parents needed to renew only for parents who renewed on time. 
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FIGURE ES.2 

REDETERMINATION OUTCOMES BY ROUND 

INDIANA OFFICE OF EARLY CHILDHOOD AND OUT-OF-SCHOOL LEARNING 
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that parents would only scan the top of the page for the appointment date and time and disre- 
gard the rest of the text. 

• The packet provided more information (including additional forms for parents who were 
self-employed or who changed jobs) in an effort to further clarify the employment verification 
requirements. 

• The team created a reminder postcard that was personalized with the client’s appointment 
time and date. 

Test — Round 2 

The Round 2 test, in which the team sent intervention materials from July to November 2014, found that 
the BIAS materials: 

• Increased the percentage of parents who attended their first scheduled redetermination ap- 
pointment by 10.6 percentage points, a 24 percent change 

• Did not increase the percentage of parents who completed the redetermination process in one 
appointment, and did not have an effect on the reasons why parents failed to renew in a given 
appointment 

• Increased the percentage of parents who renewed on time by 2.7 percentage points, a 4 per- 
cent change 

The interventions in the two rounds achieved different effects. The main success of the BIAS outreach in 
Round 1 was that it increased the percentage of parents who renewed their subsidies in one appointment, 
making the process more efficient for this group. It did not, however, change the likelihood that parents 
would renew on time because about the same proportion of parents in the control group renewed their 
subsidies, albeit by attending more appointments. In Round 2, the intervention significantly increased 
the percentage of parents who attended their first scheduled appointment and the percentage of those 
who completed redetermination on time, compared with those in the control group. The difference here 
is that there was no effect on completing the redetermination process in one appointment. This lack of 
effect may be because the intervention was able to increase the number of parents who participated in 
the redetermination process. The analysis found that the BIAS intervention decreased the percentage of 
parents who did not attend any redetermination appointments by 3.4 percentage points, or 19 percent. Not 
attending any appointments results in a loss of benefits, unless the parent appeals. Parents who would not 
otherwise attend any appointments may have more challenging life circumstances. For them, the goal of 
renewing in one appointment may be less important than starting the process by attending an appoint- 
ment and thereby improving their chances of keeping their subsidies. 

Next Steps 

These BIAS interventions focused on two steps in the CCDF child care process: choosing a child care pro- 
vider and renewing benefits. In both cases, the BIAS interventions altered existing communications and 
organizational resources to design interventions that could be implemented quickly and inexpensively. 

The interventions’ effects demonstrate that changes in existing practices informed by behavioral science 
can improve outcomes. Future research should build on these efforts by focusing on interventions that are 
more intensive, such as changing the supply of quality-rated child care providers or streamlining adminis- 
trative processes for large groups of parents. 

Behavioral economics provides a new way of thinking about the design of human services programs 
and a potentially powerful set of tools for improving program outcomes. The BIAS project offers the oppor- 
tunity for continued hypothesis testing grounded in behavioral economics and takes advantage of the 
low-cost, iterative nature of rapid-cycle experimentation. In addition to work covered in previously pub- 
lished reports (see the list of previously published research at the back of this report), the BIAS project has 
forthcoming reports on an evaluation with the Washington State Division of Child Support and a synthe- 
sis of the project’s entire body of work. 
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CHAPTER 


Introduction 


The Behavioral Interventions to Advance Self-Sufficiency (BIAS) project, sponsored by the Office of Plan- 
ning, Research and Evaluation of the Administration for Children and Families (ACF) in the U.S. Depart- 
ment of Health and Human Services, is the first major opportunity to apply a behavioral economics lens 
to programs that serve poor and vulnerable families in the United States. The project, led by MDRC, 
aims to apply behavioral insights to issues related to the operations, implementation, and efficacy of 
selected programs and policies. The goal is to learn how tools from behavioral science can be used to de- 
liver programs more effectively and, ultimately, to improve the well-being of low-income children, adults, 
and families. For additional background about behavioral economics, see Box 1.1. 

This report describes a collaboration between the Indiana Office of Early Childhood and Out-of- 
School Learning (OECOSL) and the BIAS team. The OECOSL is the lead agency responsible for adminis- 
tering the Child Care and Development Fund (CCDF) in Indiana. The federal CCDF has the dual goals of 
supporting parental employment and furthering children’s development. 1 It is the largest child care sub- 
sidy program in the United States, with the bulk of CCDF funding allocated to child care subsidies for 
low-income parents who are working or in school. 2 A portion of the grant is also set aside for initiatives 
that aim to improve the quality of child care options. The Child Care and Development Block Grant Act, 
which governs the CCDF program, was recently reauthorized to strengthen health and safety require- 
ments, create more family-friendly eligibility policies, expand quality improvement efforts, and increase 
consumer education. 3 Although the BIAS project in Indiana began before the act was reauthorized, the 
tests focus on aspects of the child care process — the selection of child care providers and redetermina- 
tion procedures — that are priorities in the new law. 

Background 

The choices parents make about child care can have significant effects on children’s outcomes later in 
life. Better- quality care in early childhood is related to higher academic achievement and fewer behav- 
ioral problems. These effects persist into adolescence. 4 In addition, well-implemented preschool programs 
can produce short-term cognitive gains, which translate over the long run into higher educational attain- 
ment and wages and less criminal activity. 5 Disadvantaged children tend to benefit more than their more 
advantaged peers from high-quality early interventions. 6 While the evidence about the effectiveness of 
school-age child care programs is more mixed, some research has shown that high-quality programs 
that partner closely with schools and sustain high levels of participation can also improve academic and 
social outcomes, particularly for disadvantaged children and youth. 7 


1 U.S. Department of Health and Human Services (2015). 

2 Lynch (2014). 

3 The Child Care and Development Block Grant Act of 2014 was reauthorized on November 19, 2014; U.S. Department of Health 
and Human Services (2015). 

4 Vandell et al. (2010). 

5 Heckman (2006); Karoly (2000). 

6 Currie (2001). 

7 For example, see Little, Wimer, and Weiss (2008). 
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BOX 1.1 

BEHAVIORAL ECONOMICS 


Behavioral economics, part of the broader field of behavioral science, is the application of psychological insights to econom- 
ic models of decision making.* Innovative research in this area has shown that human decision making is often imperfect 
and imprecise. People — clients and program administrators alike — procrastinate, get overwhelmed by choices, and miss 
important details. As a result, both program administrators and participants may not always achieve the goals they set for 
themselves. Principles from behavioral economics can both shed light on decision making and offer new tools to improve 
outcomes for program participants. 

Research has shown that small changes in the environment can facilitate desired behaviors; planning and commitment 
devices can be used to improve self-control; and default rules can produce positive outcomes even for people who fail to 
act. Over the past decade, behavioral economics has gained popularity in a variety of fields. For example, political orga- 
nizations have rapidly adopted practices informed by behavioral science. One study found that phone calls encouraging 
individuals to make plans for voting (for example, by asking when they would vote) increased voter turnout. 1 In human 
services, researchers in the UK used text message reminders delivered via cell phones to increase the percentage of gov- 
ernment benefit claimants who attended employment opportunity events.* The BIAS team previously partnered with the 
Los Angeles Department of Public Social Services to increase the percentage of Temporary Assistance for Needy Families 
recipients who reengaged in the county’s welfare-to-work program. The BIAS team designed and evaluated two forms 
of behaviorally informed, low-cost outreach to participants and found that the outreach increased positive reengagement 
with the program by 3.6 percentage points as of 30 days after an initial reengagement appointment. § 

These examples are some of the recent applications of behavioral economics to human behavior. Behavioral tweaks — or 
“nudges,” as they are frequently called — are often meant to be limited in scope. As the prominent psychologist Daniel 
Kahneman states, behavioral economics is “characterized by achieving medium-sized gains by nano-sized investments.’’^ 
These types of interventions are not always expected, or intended, to achieve enormous impacts or attain a system 
overhaul. Instead, they are meant to be responsive to behavioral tendencies and to foster change at relatively low cost and 
effort. For a more detailed overview of behavioral economics, see Behavioral Economics and Social Policy: Designing In- 
novative Solutions for Programs Supported by the Administration for Children and Families # 


For an overview of behavioral science, see Kahneman (2011). 
+ Nickerson and Rogers (2010). 

*Sanders and Kirkman (2014). 

§ Farrell, Smith, Reardon, and Obara (2016). 

Hsingal (2013). 

# Richburg-Hayes (2014).#Richburg-Hayes et al. (2014). 


While it would seem to be in the interest of both families and society for parents to use their CCDF 
subsidies to pay for high-quality child care, there is no mandate that parents do so. Moreover, they may 
not know how to assess the quality of available care. Across the country, parents can use their child 
care subsidies at any provider that meets basic health and safety standards and accepts the state’s 
reimbursement rate (which is often lower than the market rate). 8 The program does not require states 
to use their provider eligibility criteria to steer parents toward certain types of care. 9 At the same time, 
the market for child care is diverse and can be confusing to navigate. Even though child care resource 
and referral services are available nationwide to help parents understand their options, they are not the 
primary source of information for parents seeking child care. 10 Parents are more likely to rely on recom- 


8 For more information on the CCDF's below-market reimbursement rates and past research suggesting lower rates affect the 
quality of care to which parents have access, see Rohacek (2012). 

9 States have the option to pay different amounts to different types of providers. Tiered reimbursement may effectively 
reduce the supply of certain provider types, since provider types that meet higher quality standards tend to receive higher 
reimbursement. Just before the BIAS provider quality test began, Indiana introduced tiered reimbursement rates tied to the 
state's quality-rating system. Indiana Family and Social Services Administration (2014). 

10 Chase and Valorose (2010); National Survey of Early Care and Education Project Team (2014). 
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mendations from family and friends when making child care decisions, and most feel they have very few 
options. 11 

Indiana has responded to the need to give parents more information about provider quality by creat- 
ing the Paths to QUALITY (PTQ) program. PTQ is a quality-rating and improvement system that ranks 
providers on a four-point scale based on their achievement of standards related to health and safety, staff 
qualifications, parental engagement, and curriculum development. 12 PTQ is meant to provide parents 
with a simple indicator of provider quality to make the selection of quality child care easier. There are 
limits to the rating system — it applies only to providers who have volunteered to be assessed, and it 
is difficult to explain the difference in quality standards between PTQ levels — but the state believes 
that parents would generally be better off selecting a quality-rated provider than attempting to assess 
the quality of unrated providers on their own. Thus, the OECOSL wanted to use behavioral insights to 
increase the percentage of CCDF parents who selected a quality-rated provider at the time of enrollment, 
and to increase the proportion of the highest-rated providers (Levels 3 and 4) selected within this group. 

Indiana also wanted to improve its CCDF redetermination procedures. Parents must verify their 
continued eligibility for child care subsidies on a regular basis by submitting certain forms of documen- 
tation. While periodic redetermination is necessary, parents often struggle to comply with this require- 
ment. Parents receiving CCDF subsidies have reported challenges related to inefficient office procedures; 
short notice of redetermination appointments and short intervals between redetermination deadlines; 
difficulty obtaining the needed paperwork, especially employment verification documents; and little 
telephone-based support when they had questions. 13 Difficulties during the redetermination process can 
threaten the continuity of care, which, in turn, may negatively affect children’s development and parents’ 
employment. 14 These problems also translate into inefficiencies for agency staff. In Indiana, the offices 
became very congested around the time that CCDF subsidies were about to expire as parents scrambled 
to get their paperwork in on time. 

In Indiana, families had to complete the redetermination process at least once every six months 
during the time period of the BIAS study. 15 The OECOSL and its CCDF contractor for Marion County, 
Children’s Bureau, Inc., recognized that parents often had to attend multiple appointments to complete 
a single round of redetermination. The OECOSL wanted to use behavioral insights to encourage parents 
to attend their first scheduled appointment and help parents complete the process in one appointment. 

If successful, these interventions might also have increased the percentage of parents who completed 
redetermination on time. 

This report is divided into four chapters. Chapter 2 describes the BIAS intervention that aimed to 
increase the selection of quality-rated care among CCDF parents across the state. Chapter 3 describes 
two iterative interventions that aimed to make the redetermination process more efficient in Marion 
County, a large urban area that includes the city of Indianapolis. Chapter 4 situates both studies in the 
larger context of child care policy and draws broader conclusions about the three interventions. 

Chapters 2 and 3 review how the BIAS team applied an approach called “behavioral diagnosis and 
design.” As depicted in Figure 1.1, this process comprises four phases and is ideally iterative. It begins 
by defining the outcome that the team is trying to achieve in a neutral, measurable way — in this case, 


11 Forry et al. (2013). Past research found that about half or more of low-income parents considered only one option. Almost a third 
believed they had only that one option, and the vast majority (83 percent) of those who believed they had options reported only 
having one or two alternatives. See Layzer, Goodson, and Brown-Lyons (2007). 

12 Indiana Association for Child Care Resource and Referral (2015). 

13 Sandstrom, Grazi, and Henly (2015). 

14 “The length of subsidy spells is associated with the timing of subsidy redetermination, with shorter redetermination periods 
being associated with shorter subsidy spells and subsidy spells tending to end at the time of redetermination.” See Forry, 
Daneri, and Howarth (2013), p. 7. 

15 The reauthorized Child Care and Development Block Grant extends the period of eligibility to 12 months and directs states 
to ensure that their procedures do not unduly disrupt parents’ employment. These provisions aim to make the process more 
efficient. The BIAS study had a similar objective. 
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FIGURE 1.1 

THE BEHAVIORAL DIAGNOSIS AND DESIGN PROCESS 
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SOURCE: This figure was adapted from a figure created by ideas42. 

NOTE: For a more detailed description of behavioral diagnosis and design, see Richburg-Hayes et al. (2014). 


increasing the number of parents selecting quality-rated providers and reducing the number of appoint- 
ments they attend to complete the redetermination process. During the following diagnosis phase, the 
team collects qualitative and quantitative data to identify the behavioral bottlenecks that might be caus- 
ing the problem. During the third or design phase, the team uses theories from behavioral science about 
why bottlenecks might be occurring to generate proposals for interventions. In the test phase, behavioral 
interventions are evaluated using rigorous scientific methods — ideally, randomized controlled trials. 
Once the team reviews the findings from a study, it may decide to engage in subsequent rounds of test- 
ing to try to increase the intervention’s effectiveness, starting at the diagnosis phase. Chapter 3 provides 
an illustration of this rapid-cycle experimentation. 
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CHAPTER 


CCDF Parents' Use of 
Quality-Rated Care 


The first of the three BIAS tests that were conducted in Indiana aimed to encourage more parents 
receiving Child Care and Development Fund (CCDF) subsidies to use providers who participate in the 
Paths to QUALITY (PTQ) system, and to choose providers at the upper end of the rating scale. This study 
was conducted statewide. The Office of Early Childhood and Out-of-School Learning (OECOSL) saw this 
test as a way to help parents identify and use providers that met certain quality standards. 

Child care providers in Indiana fall broadly into five types: center-based; home-based; religious 
ministry; school age; and relative, friend, or neighbor. In addition, there are three licensing categories: 
licensed, unlicensed, and legally exempt. Every type of provider except relative, friend, or neighbor is eli- 
gible to participate in PTQ. Providers volunteer for the program and are rated by the state for achieving 
certain standards. In July 2013, over 2,300 providers with capacity for over 86,000 children were partici- 
pating in PTQ across the state, including over 66 percent of licensed providers. The state has been less 
successful recruiting religious ministry providers into the system, with only 12 percent participating. 
There are four levels to the rating system, with the following standards: 

• Level 1 providers meet the health and safety standards that are required for state licensing, 1 
which means any licensed program can join PTQ and earn at least a Level 1 rating. 

• Level 2 providers provide learning materials and activities and daily reading exercises, em- 
ploy more highly trained and credentialed staff, and follow nurturing staff and organizational 
practices, such as regular, structured communication with families. 

• Level 3 providers employ even more highly trained staff, offer many diverse activities, and 
implement a curriculum aligned with the state’s standards for preparing children to succeed 
in school. 

• Level 4 providers are accredited by one of several national organizations, which Indiana 
regards as meeting the highest standards for quality early child care and education. 

From 2008 through 2011, academic researchers assessed a random, statewide sample of over 270 
PTQ providers using research-validated measurement tools. 2 They found that Level 4 providers delivered 
significantly higher-quality care and education when compared with Level 1 providers, indicating that 
PTQ can provide useful information to parents about provider quality, at least at the extremes of the rat- 
ing scale. There is no other state measure of the quality of child care providers (except licensure), and it 
is therefore not possible to make general statements about the quality of unrated providers. 

As noted in the introduction, the BIAS team used a process called behavioral diagnosis and design 
to investigate CCDF parents’ decision making about child care and their use of the PTQ system to 
evaluate the quality of providers. 


1 Providers in Indiana can be exempt from licensing requirements and still provide child care if they meet certain conditions, the 
most common of which is affiliation with a church or religious ministry. 

2 Researchers used the Environment Rating Scales and the Caregiver Interaction Scale. For more information about the 
evaluation and its findings, see Elicker et al. (2011). 
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Define 

At the time the BIAS team began discussions with OECOSL staff, about 65 percent of CCDF families 
— over 12,000 — were enrolled with PTQ providers. While this participation rate is high, OECOSL staff 
believed that there was room for improvement because of the fairly large number of PTQ-rated provid- 
ers. They also believed that many parents may not be aware of or understand the quality-rating system. 
Their data showed that CCDF families switched between rated and unrated providers in equal numbers, 
suggesting that the PTQ rating might not be a major criterion for them when selecting a provider. The 
goals of the behavioral intervention were to: 

• Increase the percentage of CCDF parents who selected a quality-rated provider 

• Encourage CCDF parents to select higher-rated providers (those with a 3 or 4 rating) 

Diagnose 

Indiana has traditionally had more demand for its CCDF subsidies than available funds, and the 
OECOSL puts families it cannot serve immediately on a waitlist. 3 The office notifies parents when a sub- 
sidy is available, and they then have a few days to select their provider, prepare eligibility documents, 
and attend an enrollment appointment. 

Before collaborating with the BIAS team, OECOSL staff realized they could market PTQ to par- 
ents on the waitlist since parents may need or want to select a new provider at the time of enrollment. 
Parents on the waitlist might not have a stable form of child care, or might be using a provider that is not 
eligible to accept the CCDF subsidy. 4 The subsidy provides parents with additional resources to pay for 
child care and allows them access to previously unaffordable options. 

The OECOSL had begun mailing parents a letter and brochure about PTQ before the BIAS interven- 
tion was launched. The letter discussed the differences in safety and training standards between rated 
and unrated providers, explained how to contact the Indiana Association of Child Care Resource and 
Referral (IACCRR) for child care referrals, and reminded parents they must have chosen a provider by the 
time of their CCDF enrollment appointment. 5 

The BIAS team investigated why some parents who enroll in the CCDF subsidy program do not 
choose a quality-rated provider. The team mapped out each step parents must take to apply for a sub- 
sidy, select a provider, and enroll. The team then spoke to a small number of parents and reviewed data 
about the CCDF program and PTQ enrollment rates, the use of the subsidies for various types of care, 
the number of families on the waitlist, and the child care referral system. 

In line with existing research, the diagnosis stage revealed several factors that generally constrain 
parents’ child care choices. 6 Two of these constraints are notable in the context of this study. The first is 
a lack of access to PTQ providers, especially those with Level 3 or 4 ratings. Of the providers that partici- 
pate in the program, only about 12 percent are Level 4 and 22 percent are Level 3. 7 It is also impossible 
to know how many of these providers have age -appropriate vacancies or how many are conveniently 
located near parents (which often means near a parent’s or grandparent’s home, or on a parent’s way to 


3 To apply, parents must submit an application form to the local intake office. Applicants must check off a box on the form that 
asks whether they are working or in school and provide a pay stub or tax form schedule if they are working. Thus, applicants 
likely have a service need when they arrive at the intake office to apply for the benefit. 

4 Providers must meet certain standards for health and safety to accept the subsidy. 

5 IACCRR oversees regional child care resource and referral offices and operates a statewide hotline. It serves as a resource to 
government, media, social service providers, and others about child care systems and resources across Indiana. 

6 Family and community characteristics can affect the choices parents have or perceive themselves to have. See Weber (2011). 

7 Data are calculated based on the January 2015 PTQ report. 
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work or school). Low-income parents, particularly those in rural areas, may have very limited options. 8 
The second prominent constraining factor is the high cost of care. Parents can face out-of-pocket 
expenses for child care even when the provider accepts a CCDF subsidy. In addition to copayments, 
which in Indiana are calculated as a percentage of income, providers may also charge parents the differ- 
ence between the amount of the subsidy and the market rate price (“overage cost”). Since higher-quality 
providers tend to be more expensive, overage costs generally increase with the quality of care, although 
individual providers may decline to charge any overage cost. 9 Addressing these factors was outside the 
scope of the BIAS project, which designed and tested inexpensive, easy-to-implement interventions. 

The behavioral diagnosis focused on issues related to parents’ awareness of the quality-rating and 
improvement system, criteria for selecting child care, and ability to choose quality-rated providers. 

These factors are behavioral and have been found to play an important role in the decision-making pro- 
cess. The behavioral obstacles — or “bottlenecks” — that the BIAS team believed to be most significant 
in hindering the selection of a PTQ provider, and their associated behavioral terms, are described below 
(and displayed in Figure 2.1). Full definitions of these terms, which appear in boldface the first time they 
appear in this chapter, are presented in Appendix Table A.l. 

Bottleneck 1: Parents may not be aware of or understand the benefits of the PTQ program. 

Parents do not receive clear information about the different standards associated with various provider 
types. Licensing and quality standards are not clearly explained, and parents may have no reference 
points for evaluating care, aside from their own experience. IACCRR can provide parents with individu- 
alized child care referrals, but parents may not be aware of the resource. The social norm for parents 
may be to search for care through social networks that they trust. 

Bottleneck 2: Parents may not begin their search for a specific child care provider while on 
the waitlist. 

The OECOSL does not provide parents with an estimate of how long they will be on the waitlist or the 
amount the subsidy will cover, and therefore parents may not initiate a search during this time. Parents 
sometimes wait months for a CCDF voucher slot to become available, and their circumstances may 
change (for example, a loss of job) in the interim. They also may not begin a search while on the waitlist 
if obtaining the subsidy seems psychologically distant. The uncertainty may make it hard to justify 
taking time out of their busy schedule to search for options, a possible result of a concept known as 
present bias. 

Bottleneck 3: Parents have a short time period to choose a provider once the OECOSL calls 
them off the waitlist, forcing them to act quickly. 

Parents have only days to choose a provider and prepare for the enrollment appointment once the 
OECOSL calls them off the waitlist. The materials sent from the intake office may involve high delibera- 
tion costs (the costs in time or mental effort of making a decision) because they contain a great deal of 
information. Parents must set aside time in their schedule to attend the enrollment appointment, gather 
all the necessary documentation, and select a provider. Under these conditions, parents who have the 
option to use the subsidy at their existing provider may choose to do so — an example of status quo 
bias. Those who cannot stay with their current child care arrangement have limited time and resources 
for selecting an eligible provider. As a result, they may show a primacy bias and select the first or sec- 
ond provider they visit. 

The BIAS team hypothesized that parents could benefit from information about quality-rated provid- 
ers delivered to them while they are on the waitlist. This information would have to be delivered before 
their CCDF enrollment appointment and in a manner that was compelling enough to motivate parents to 


8 Weber (2011); Forry et al. (2013). 

9 Minton, Durham, and Giannarelli (2014). 
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FIGURE 2.1 

BEHAVIORAL MAP FOR SELECTING A CCDF PROVIDER 

INDIANA OFFICE OF EARLY CHILDHOOD AND OUT-OF-SCHOOL LEARNING 


CCDF ENROLLMENT AND PROVIDER PROCESS BOTTLENECK 
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Reference points, social norm, trust 
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SOURCE: Figure based on BIAS behavioral diagnosis research with staff from the Indiana Office of Early Childhood and Out-of-School Learning 
and the Indiana Association for Child Care Resource and Referral. 

NOTE: This map is a stylized representation of the child care provider selection process. A parent does not necessarily go through all these steps 
in the displayed order. 


begin their search while on the waitlist. It would also have to be simple so parents could better under- 
stand and weigh their complex child care choices. 


Design 

The main component of the behaviorally informed intervention was individualized child care referrals 
delivered at a moment when parents were likely thinking about child care options . 10 To implement it, the 
BIAS team turned to IACCRR, the statewide child care resource and referral agency. IACCRR had the 
existing capacity to deliver free, individualized child care consumer education and referrals to fami- 
lies. The BIAS intervention integrated this capacity into the CCDF application process in a new way by 
initiating contact with parents on the waitlist rather than only responding to requests for information 


10 Past research found that providing families with simplified information about the test scores of the available school options 
caused significantly more parents to choose higher-scoring schools for their children. See Hastings and Weinstein (2008). 
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TABLE 2.1 

HYPOTHESIZED RELATIONSHIPS OF BOTTLENECKS, BEHAVIORAL CONCEPTS, 
AND COMPONENTS OF THE INTERVENTIONS - PTQ TEST 

INDIANA OFFICE OF EARLY CHILDHOOD AND OUT-OF-SCHOOL LEARNING 


INTERVENTION COMPONENTS 3 


HYPOTHESIZED BOTTLENECK AND 

BEHAVIORAL CONCEPTS 

Personalization 

Simplification 

Implementation 

Prompts 

Proactive Human 

Outreach 

PARENTS MAY NOT BE AWARE OF OR UNDERSTAND THE BENEFITS OF THE PTQ PROGRAM. 

Reference points 





Social norm 





Trust 





PARENTS MAY NOT BEGIN A SEARCH FOR CHILD CARE WHILE ON THE WAITLIST. 

Psychological distance 





Present bias 





PARENTS HAVE A SHORT TIME PERIOD TO CHOOSE A PROVIDER ONCE THEY ARE CALLED OFF THE WAITLIST, FORCING 
THEM TO ACT QUICKLY. 

Deliberation costs 





Status quo bias 





Primacy bias 






NOTES: Behavioral concepts cannot be definitively identified, but rather are hypotheses derived from the behavioral diagnosis and design 
process that may explain behavioral bottlenecks. 

^he following are examples of intervention components in Indiana: 

Personalization: Provide individualized referrals. 

Simplification: Streamline the information provided about the PTQ program. 

Implementation prompts: Provide parents with spaces on the cover letter to list their child care provider selection. 

Proactive human outreach: Staff members call parents to offer personalized assistance. 


from parents as IACCRR usually does. IACCRR provided individualized referrals in a redesigned packet 
of information that highlighted the quality-rated providers near the applicant’s home. To further facilitate 
the decision-making process, IACCRR staff reached out to a subset of parents by phone. 

The Revised Packet 

The BIAS team replaced the letter and brochure typically sent to parents with a packet designed using 
principles from behavioral science. These materials aimed to inform parents about the benefits of PTQ 
and encouraged them to begin their search for providers right away. The BIAS packet contained the be- 
havioral elements of personalization, simplification, implementation prompts, and visual cues 

described below. (Table 2.1 connects hypothesized bottlenecks, behavioral concepts, and intervention 
components.) 

• A redesigned letter (page 1). The cover letter was personalized, beginning with the par- 
ent’s name handwritten by IACCRR staff, along with a handwritten note at the bottom with 
the name of the staff person who selected the referrals (Figure 2.2). The BIAS team chose this 
design to catch parents’ attention and increase the likelihood that they would continue read- 
ing about the PTQ program, addressing Bottleneck 1. Even if the packet was successful at 
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FIGURE 2.2 

SELECT EXCERPTS FROM THE BIAS PTQ PACKET 

INDIANA OFFICE OF EARLY CHILDHOOD AND OUT-OF-SCHOOL LEARNING 
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2. Check out the list of providers located near your home. We have included a list of 
providers that serve your child's age group to help you start your seard. 
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FIGURE 2.2 

(CONTINUED) 


A potential child care provider for: Reference Number: 

We have provided this information to help you find child care that meets your family’s needs. This referral is a 
starting point for your search, but we do not recommend or license specific programs. 

If you need more help, call us at\ -800-299-1 62 or come to our website: www.childcareindiana.org. 
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PROVIDER QUALITY 

Paths to QUALITY Level: Paths To QUALITY Level 1 
License Type: Licensed, Class I 

PROVIDER FEATURES 
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convincing parents to consider a PTQ provider, behavioral research demonstrates that when 
people say they want to do something, they do not always follow through on their intentions. 
To assist parents in plan making, the letter used an implementation prompt to guide them 
through the search for child care. 11 These step-by-step instructions were designed to urge 
parents to begin their search while on the waitlist, addressing Bottlenecks 2 and 3. 

• An explanation of PTQ (page 2). A graphic was added to explain the benefits of PTQ, 
including the four levels of quality. This visual cue described the benefits of choosing a PTQ 
provider in simple terms and indicated the additional benefits at each higher level. A simpli- 
fied checklist also presented many reasons to select a PTQ provider. These design elements 
aimed to communicate the benefits of the program clearly and concisely, addressing Bottle- 
neck 1. 

• A customized map and three personalized referrals of providers near the parent’s 
home (pages 3-7). Created with software that IACCRR programmed for the BIAS interven- 
tion, this component of the package was highly personalized. Similar to Google Maps, a map 
displayed the location of three providers near the parent’s home and included important in- 
formation about each of them (PTQ rating, name, and phone number). Behaviorally designed 
individual referral pages provided more information about the providers and were arranged 
from the highest PTQ rating to unrated. The BIAS team generated the referrals based on the 
parent’s home address and a match between the age of children served and the needs of the 
family, though it was not possible for IACCRR to filter out providers that were at capacity 

for the age group relevant for each family due to data limitations. 12 Referrals were limited to 
three per mailing to avoid information overload. This element of the packet aimed to help 
parents recognize that PTQ providers were within reach and, by showing the proximity of 
the providers to their home, encourage them to visit the providers while they were still on the 
waitlist (addressing Bottlenecks 1 and 2). 

• A parent’s guide to visiting a provider (page 7). The last page in the packet presented 
a checklist of questions for parents to ask when visiting a potential child care provider. It was 
intended as a resource and prompt for parents to begin their search, addressing Bottlenecks 2 
and 3. 

Personal phone calls from IACCRR staff 

• Some parents who were sent a BIAS packet also received a personal phone call from an 
IACCRR customer education staff member. This outreach was intended to build trust and 
overcome the status quo bias to stay with the same child care provider. The phone call aimed 
to both improve parents’ knowledge about the PTQ program (addressing Bottleneck 1) and 
encourage parents to begin their search while on the waitlist (addressing Bottlenecks 2 and 
3). IACCRR staff asked parents if they received the packet, if they had any questions about 

it, and if they needed additional referrals. The staff member began the call with a loose script 
developed by the BIAS team, but was given flexibility to answer a range of questions, with 
the goal of fully informing parents about their options in selecting a child care provider. 


11 Implementation prompt is a common term for a strategy designed to encourage individuals to make detailed plans to fulfill 
goals. See Milkman et al. (2011) and Milkman et al. (2012). 

12 In most cases, the providers were within a 5 mile radius of the parent’s home address on record. In some rural areas where 
there was no PTQ provider within 5 miles, the BIAS team expanded the radius to 10 miles if that would have included at least 
one PTQ provider. 


CUTTING THROUGH COMPLEXITY 12 


Test 


Intervention materials were mailed to families from June to October 2014; the BIAS team tracked parents’ 
child care choices for those who enrolled in the CCDF program for eight additional months. 13 There were 
three study groups: a control group that received the standard OECOSL letter and brochure, an intervention 
group that received personalized referrals by mail (referrals-only group), and another intervention group that 
received personalized referrals and a phone call from IACCRR staff (referrals-phone group). The evaluation 
estimated the effects of the referrals-only and referrals-phone groups, compared with the control group, 
on the following primary outcomes: 

• Selection of a PTQ provider at the time of enrollment 

• Selection of a highly rated PTQ provider (Level 3 or 4) at the time of enrollment 

At the beginning of each week, the OECOSL sent the BIAS team an anonymized list of parents 
whose children had been added to the CCDF waitlist the week before. The BIAS team randomly 
assigned them to one of the three study groups and forwarded the list to IACCRR. 14 Its staff sent out all 
mailings within 24 hours, and attempted the first phone calls within a week. It was a significant tech- 
nological and organizational achievement for IACCRR to create profiles for each family in the sample, 
replace their regular referral materials with those designed by the BIAS team, and conduct personal 
phone calls within the specified timeframe. 

Appendix Figure A.l depicts the randomization process used in the PTQ intervention (as well as the 
redetermination intervention discussed in Chapter 3). Parents were randomly assigned, with all their 
children assigned to the same study group, since parents were the ones who were mailed the interven- 
tion materials and make decisions on behalf of their children. The sample included 12,652 children, and 
children are the unit of analysis because a family with multiple children may have different child care 
arrangements for each one. The effects of the intervention are sometimes reported below on “families” to 
recognize that parents received the intervention and made decisions about child care. 

In the first week of the intervention, the BIAS team randomly assigned the parents of all 5,949 chil- 
dren who were currently on the CCDF waitlist to the three study groups and forwarded the assignments 
to IACCRR. These groups make up Cohort 1. Beginning in the second week of the intervention, the team 
randomly assigned parents to the three study groups on a rolling basis within about a week of them 
signing up for the waitlist. These groups make up Cohort 2 and include 6,703 children. 

The cohorts differ in two ways that have implications for the analysis. First, due to limitations in 
the state’s data, it was not possible to determine exactly when parents in Cohort 1 signed up for the 
CCDF waitlist. However, given the size of the group, it can be assumed that many families had been on 
the waitlist for months. Thus, unlike parents in Cohort 2 who were sent intervention or control materi- 
als close to when they signed up for the waitlist, many parents in Cohort 1 received the materials after 
a substantial delay. This difference provides an opportunity to broadly examine whether impacts vary 
based on the timing of the intervention, relative to when families sign up for the waitlist. Second, the 
random assignment ratios are not equal in the two cohorts. Because the BIAS team randomly assigned 
Cohort 1 in one week, it limited the number of parents assigned to the referrals-phone group to ensure 
IACCRR staff could conduct all the personal phone calls within the intended timeframe. As a result, 
there are 4,255 children in the control group, 1,191 in the referrals-only group, and 503 in the referrals- 
phone group. Cohort 2 includes about an equal number of children in each study group: 2,402 in the 


13 This intervention was originally launched on February 3, 2014. Two weeks into implementing it, the state received an influx of 
CCDF funding due to a surplus from Temporary Assistance for Needy Families. While there are usually more parents in need of 
subsidies than can be served, that was not the case at the time and the waitlist was eliminated. With no gap in time between 
when parents expressed an interest in the CCDF program and when they enrolled, there was no opportunity to deliver 
information to parents before their enrollment appointment and the BIAS study was suspended. In June 2014, the state notified 
the BIAS team that it had spent the additional funds and would reopen the waitlist. There was some uncertainty about the 
number of parents who would apply for a CCDF subsidy during the intervention period because many more than usual had 
been served in the prior months; however, the study was relaunched on June 16, 2014. 

14 The team randomly assigned all children in a family to the same study group. 
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control group, 2,389 in the referrals-only group, and 1,912 in the referrals-phone group. Since Cohort 2 is 
more balanced, it provides a more precise estimate of the intervention’s true effects. The two cohorts are 
treated as subgroups in the following analysis. 

Findings 

This section presents findings on the test’s implementation, its main effects on the full sample, and its 
effects on the cohort subgroups. 15 

Implementation 

• The interventions demonstrate that it is feasible for personalized child care infor- 
mation to be delivered to applicants while they are on the CCDF waitlist. 

The study ran as expected, affirming the feasibility of integrating the CCDF subsidy program and 
child care resource and referral processes. While there were some start-up expenses related to transfer- 
ring data and mailing the referral lists, IACCRR had the existing capacity to create the referrals and 
make phone calls for a statewide intervention without increasing the number of staff. 

Based on analysis of data from IACCRR’s management information system (not shown), almost all 
parents (an average of 97 percent) were mailed the materials they were meant to receive, and IACCRR 
mailed most (95 percent) of the materials a week before parents’ CCDF enrollment appointment for fami- 
lies whose children actually enrolled in the CCDF program. Over 96 percent of parents in the two inter- 
vention groups combined received referrals to only quality-rated providers. Most providers listed on the 
referral sheets were Level 1 because most of the providers who participate in PTQ are Level 1. However, 
nearly 70 percent received at least one referral to a Level 3 or Level 4 provider. 

Consumer education staff attempted to call 98 percent of parents in the referrals-phone group, of 
whom 57 percent participated in a call. In the rest of the cases, IACCRR staff left a voicemail message or 
otherwise could not speak to the parent, or the number was out of service. Of parents who spoke to a staff 
person, 30 percent asked for more referrals. The top reasons parents cited for needing more referrals were 
to learn more about their options and to get referrals to providers with different hours or in different areas. 

• Fewer than half of children on the CCDF waitlist were enrolled in the CCDF pro- 
gram during the 12-month study period. 

About 43 percent of children on the waitlist were enrolled in the CCDF program by the end of the 
study period. The rest — more than 7,000 children — were not enrolled during the study period. (See 
Table 2.2.) It is not possible to know if the main reason most families did not receive CCDF subsidies 
was because (1) of insufficient funding, (2) they were not eligible at the time they were offered the sub- 
sidy, (3) their child care needs changed, or (4) they intended to enroll but did not complete the process 
because of behavioral barriers or administrative hassles. Many families spent a fairly long time on the 
waitlist, and thus their needs or motivations could have changed. 

From an analytical perspective, the low rate of enrollment means that the BIAS team could not 
collect child care provider selection data for the majority of the sample. In line with conventional evalu- 
ation methods, the team treated families that never enrolled in the CCDF program as unresponsive to 
the behavioral interventions. However, if the team had excluded individuals for whom there was no child 
care provider selection data from the sample or assumed the outcomes for them to have been the same 
as those for families that were enrolled in the CCDF program, the impacts described below would be 
larger. 


15 More information about the research analyses, including the estimation equation, weighting to adjust for differences in random 
assignment ratios between cohorts, and exploratory analyses can be found in the Technical Appendix (Appendix B). 
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TABLE 2.2 

PROVIDER SELECTION OUTCOMES, PTQ TEST 

INDIANA OFFICE OF EARLY CHILDHOOD AND OUT-OF-SCHOOL LEARNING 


OUTCOME 

CONTROL 

GROUP 

REFERRALS- 

ONLY 

GROUP 

DIFFERENCE 

REFERRALS- 

PHONE 

GROUP 

DIFFERENCE 

OVERALL 

AVERAGE 

Enrollment 

Enrolled in the CCDF program (%) 

42.0 

42.3 

0.2 

44.7 

2.6 

42.6 

Provider quality-rating selection 

Selected an unrated provider (%) 

13.4 

14.4 

1.0 

14.2 

0.8 

13.8 

Selected a PTQ provider (%) 

28.7 

27.9 

-0.7 

30.4 

1.8 

28.8 

Level 3 or 4 provider 

12.6 

13.2 

0.6 

14.7 

2.1* 

13.1 

Level 4 provider 

6.0 

6.7 

0.8 

8.0 

2.0** 

6.6 

Level 3 provider 

6.6 

6.4 

-0.2 

6.8 

0.2 

6.6 

Level 2 provider 

2.9 

3.3 

0.4 

2.5 

-0.5 

2.9 

Level 1 provider 

13.2 

11.5 

-1.7* 

13.3 

0.1 

12.7 

Sample size (total = 12,652) 

6,657 

3,580 


2,415 




SOURCE: MDRC calculations based on data from the Office of Early Childhood and Out-of-School Learning. 

NOTES: Estimates were regression-adjusted using ordinary least squares, controlling for pre-random assignment characteristics of 
sample members. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Rounding may cause slight discrepancies in sums and differences. 


Impacts 

• The interventions did not increase the percentage of enrolled families who chose 
PTQ providers, but it shifted the preferences of parents who used quality-rated 
providers in both intervention groups. The referrals-phone intervention led to a 
moderate increase in the percentage of families choosing highly rated providers 
(Levels 3 and 4), while the referrals-only intervention led to a modest decrease in 
the percentage choosing Level 1 providers. 

Table 2.2 shows the percentage of each group that selected an unrated provider or a PTQ provider. 
The table further lists the quality rating of the providers selected and whether the provider was a Level 3 
or 4 (highly rated). 

The behavioral interventions did not change the percentage of families who chose an unrated pro- 
vider, compared with a PTQ provider. About one-third used unrated providers and two-thirds used PTQ 
providers. 

The referrals-phone intervention increased the selection of highly rated providers (Level 3 or 4) by 
2.1 percentage points. The increase was almost entirely due to a 2.0 percentage point increase in the 
selection of Level 4 providers, a 33 percent change. The referrals-only intervention slightly decreased the 
percentage of families who chose a Level 1 provider by 1.7 percentage points, a 13 percent change. There 
were no other changes in the choices parents made regarding providers’ quality rating. 

Subgroup Analysis 

• The effectiveness of the behavioral interventions differed by cohort. 

The subgroup analysis in Table 2.3 shows that the referrals-only intervention had an impact on fami- 
lies choosing a Level 4 provider in Cohort 2, but not Cohort 1. The referrals-only intervention increased 
the percentage of families who chose a Level 4 provider in Cohort 2 — those who received the refer- 
rals shortly after signing up for the waitlist — by 1.8 percentage points or 30 percent, but had no effect 
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on this outcome in Cohort 1. This finding suggests that receiving referrals close to when parents sign 
their children up for the waitlist may impact whether they decide to use the highest-rated providers. 
(Appendix Figure A. 2 depicts the selection of highly rated providers by study group.) The referrals-only 
intervention in Cohort 2 had two other noteworthy statistically significant effects, but those effects were 
not statistically significantly different from the lack of effects in Cohort 1 (signified in the table by the 
absence of a t symbol), and thus must be treated with caution. The referrals-only intervention increased 
the percentage of families who chose an unrated provider and decreased the percentage who chose a 
Level 1 provider. The referrals may have influenced parents who would have otherwise selected a Level 
1 provider for their children, encouraging some to choose higher-rated providers and others to opt out of 
the PTQ system completely. 

Table 2.3 helps explain the main impacts of the referrals-phone intervention on the full sample. There 
is no statistically significant difference in impacts between the two cohorts for that intervention, but this 
lack of difference is likely because of the highly imbalanced sample sizes in Cohort 1, in which only 503 
families received phone calls. In Cohort 2, on the other hand, the referrals-phone intervention increased 
the percentage of families who selected a Level 4 provider by 2.7 percentage points, and the percentage 
who chose a highly rated (Level 3 or 4) provider by 4.0 percentage points. 

These findings suggest that delivering the referrals-phone intervention close to when parents sign up 
for the CCDF waitlist may be a promising area for future research. The effects reported here are diluted 
because about 60 percent of families did not enroll in the CCDF program. Since there is no reliable informa- 
tion about the child care choices these families made, they are treated as unresponsive to the interven- 
tion. In a context where a larger proportion of waitlisted children enroll in the CCDF program, it would be 
possible to obtain a more precise estimate of the impact of the behavioral interventions. Appendix Table 
B.l shows enrollment levels and provider selection for the subset of families who signed up for the CCDF 
program during the follow-up period and received the intervention close to when they signed up for the 
waitlist (Cohort 2) and serves as an illustration of what the proposed replication might find. This analysis 
is nonexperimental because children who were not enrolled in the CCDF program have been removed, and 
thus there are no significance tests. 


Discussion 

Choosing child care is a complex activity. It involves weighing structural factors related to quality and 
accessibility (such as resources, hours of operation, and location) and process factors that may be harder 
to evaluate (such as staff’s warmth or parental engagement). Once parents have decided on a provider, 
they must follow up with the appropriate paperwork to finalize their selection. For low-income parents 
enrolling in a CCDF program with a waitlist, they do much of this follow-up without knowing for cer- 
tain when they will actually receive the subsidy, and their time and attention are split between work 
or school and family commitments. In this context, it is meaningful that these behavioral interventions 
affected parents’ decision making about child care. 16 

The behavioral interventions tested in this study did not change the likelihood that families would 
choose a PTQ provider (which was high to begin with), but they did shift preferences among those who 
chose a quality-rated provider. In the full sample, families in the referrals-only group were slightly less 
likely to choose a Level 1 provider. There were no other statistically significant changes in the choices 
these families made, but it is reasonable to assume that parents who shifted away from Level 1 providers 
chose higher-rated or unrated providers instead. 

The decision to select a Level 2 or higher provider over a Level 1 provider is not surprising given the 
outreach emphasizing the benefits of programs with higher ratings. The decision to choose an unrated 
provider over a Level 1 provider is also unsurprising in some situations. Parents choosing between a Level 
1 and an unrated provider may have opted out of the PTQ system because of the general aversion people 


16 Forry et al. (2013). 
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TABLE 2.3 

PROVIDER SELECTION OUTCOMES BY COHORT SUBGROUP, PTQ TEST 


INDIANA OFFICE OF EARLY CHILDHOOD AND OUT-OF-SCHOOL LEARNING 

PANEL 1: REFERRALS-ONLY VERSUS CONTROL 


OUTCOME 

REFERRALS- 

ONLY 

GROUP 

COHORT 1 

CONTROL 

GROUP 

DIFFERENCE 

REFERRALS- 

ONLY 

GROUP 

COHORT 2 

CONTROL 

GROUP 

DIFFERENCE 

ACROSS- 

COHORT 

SIGNIFICANCE 

Enrollment 








Enrolled in the CCDF program (%) 

39.0 

40.6 

-1.6 

45.1 

43.3 

1.9 


Provider quality-rating selection 








Selected an unrated provider (%) 

13.0 

13.6 

-0.6 

15.6 

13.2 

2.4* 


Selected a PTQ provider (%) 

26.1 

27.1 

-1.0 

29.5 

30.0 

-0.5 


Level 3 or 4 provider 

12.1 

12.3 

-0.2 

14.1 

12.8 

1.3 


Level 4 provider 

5.5 

6.0 

-0.4 

7.8 

6.0 

* 

°o 

i 

t 

Level 3 provider 

6.6 

6.3 

0.3 

6.3 

6.8 

-0.6 


Level 2 provider 

2.5 

2.5 

0.0 

4.0 

3.3 

0.7 


Level 1 provider 

11.5 

12.3 

-0.8 

11.5 

13.9 

-2.5* 


Among children enrolled (%) 








Selected an unrated provider 

33.2 

33.4 

- - 

34.4 

30.6 

- - 

- 

Selected a PTQ provider 

66.8 

66.6 

- - 

65.6 

69.4 

- - 

- 

Sample size (total = 10,237) 

1,191 

4,255 


2,389 

2,402 




PANEL 2: REFERRALS-PHONE VERSUS CONTROL 




COHORT 1 



COHORT 2 



OUTCOME 

REFERRALS- 

PHONE 

GROUP 

CONTROL 

GROUP 

DIFFERENCE 

REFERRALS- 

PHONE 

GROUP 

CONTROL 

GROUP 

DIFFERENCE 

ACROSS- 

COHORT 

SIGNIFICANCE 

Enrollment 




Enrolled in the CCDF program (%) 

43.3 

40.6 

2.6 

46.0 

43.3 

2.8 


Provider quality-rating selection 




Selected an unrated provider (%) 

16.3 

13.6 

2.8 

12.4 

13.2 

-0.8 


Selected a PTQ provider (%) 

27.0 

27.1 

-0.1 

33.6 

30.0 

CO 

00 


Level 3 or 4 provider 

12.4 

12.3 

0.1 

16.8 

12.8 

4 Q*** 


Level 4 provider 

7.2 

6.0 

1.3 

8.7 

6.0 

2.7*** 


Level 3 provider 

5.2 

6.3 

-1.2 

8.2 

6.8 

1.3 


Level 2 provider 

1.4 

2.5 

-1.2* 

3.4 

3.3 

0.1 


Level 1 provider 

13.2 

12.3 

0.9 

13.4 

13.9 

-0.5 


Among children enrolled (%) 




Selected an unrated provider 

37.7 

33.4 

- - 

27.0 

30.6 

- - 

- 

Selected a PTQ provider 

62.3 

66.6 

- - 

73.0 

69.4 

- - 

- 

Sample size (total = 9,072) 

503 

4,255 


1,912 

2,402 




SOURCE: MDRC calculations based on data from the Office of Early Childhood and Out-of-School Learning. 

NOTES: Estimates were regression-adjusted using ordinary least squares, controlling for pre-random assignment characteristics of sample members. 
Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Differences across cohort impacts were tested for statistical significance. Those statistical significance levels are indicated as follows: ttt = 1 percent; 
tt = 5 percent; t = 10 percent. 

Rounding may cause slight discrepancies in sums and differences. 

Characteristics shown in italics are calculated for a portion of the full sample and indicate nonexperimental data. Significance tests are not calculated 
for nonexperimental data. 
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have to being in the last position on an ordinal scale. 17 In addition, parents in this situation may find that 
the unrated provider is better suited for them. Unrated providers are not necessarily of poorer quality than 
PTQ providers. When the unrated provider is a more informal type of care, it may offer advantages such as 
flexible hours, lower cost, or cultural affinity. If the unrated provider is licensed, it has achieved the mini- 
mum standards for Level 1, which might make it hard for parents to distinguish any differences in quality. 

The combination of referrals and a personal phone call increased the proportion of families who 
selected higher-rated providers, especially Level 4 providers, by a statistically significant margin. Parents 
who have the option to choose a Level 4 provider may be in a less ambiguous situation when it comes to 
weighing their options. The differences in quality between, for example, Level 1 and Level 4 are easy for 
IACCRR staff to articulate and should be easy for parents to see in the providers’ structural characteris- 
tics, such as the facilities, resources, and staff. The behavioral outreach may have helped some parents 
understand that they can access this sort of high-quality care with the CCDF subsidy. Program supply, 
however, remains a challenge; most parents (88 percent) received at least one referral to a Level 1 pro- 
vider, while only 34 percent received at least one referral to a Level 4 provider. 

This study adds to existing research on the effects of personalized information on decision mak- 
ing. 18 It also sheds light on the value of personal assistance and how the timing of information can influ- 
ence decisions. 

The BIAS team was not able to interview parents in the sample, and so it is not possible to fully 
explain how the different forms of outreach affected their decision making. The behavioral interventions 
aimed to create an implementation prompt for parents to use quality-rated providers if given the chance. 

It is possible that some parents used the referrals in the packet to locate nearby child care providers. But 
given how difficult it can be to match child care providers to families’ needs, it is also possible that the 
behavioral interventions helped some parents make a plan for finding a suitable child care provider. This 
scenario might explain why receiving a phone call increased the selection of highly rated child care more 
than referrals alone. IACCRR program data indicated that, on average, parents in the referrals-phone group 
had 1.5 interactions with consumer education staff, suggesting that some parents called IACCRR back 
after receiving the initial call, perhaps when they were called off the waitlist. For these parents, IACCRR 
may have become a trusted resource for overcoming the informational and motivational barriers to select- 
ing a child care arrangement that was different from what they would have otherwise chosen. 

The behavioral interventions worked only when delivered soon after families signed up for the waitlist 
(Cohort 2), even though they likely could not enroll in the CCDF program until weeks or months later. This 
finding contradicts common assumptions about how timing relates to decision making and action. The 
optimal time for an intervention is typically thought to be close to when people will perform the desired 
action unless the intervention includes some automatic escalation or initiation component. In this case, 
the intervention was more effective when delivered closer to when parents signed up for the waitlist than 
when they were called off of it. Future research might explore whether the effects of this intervention could 
be increased by reaching out to parents again around the time they receive the offer to enroll in the CCDF 
program. 


17 Kuziemko, Buell, Reich, and Norton (2014). 

18 Hastings, Van Weelden, and Weinstein (2007); Hastings and Weinstein (2008). 
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CHAPTER 


Renewing CCDF Subsidies 


At the time of the BIAS project, parents in Indiana had to prove their continued eligibility for the Child 
Care and Development Fund (CCDF) subsidy every six months by providing certain forms of documenta- 
tion. 1 The state of Indiana establishes the CCDF redetermination policies, but contractors in each county 
determine how to implement the procedures. The tests described in this chapter took place in Marion 
County, Indiana, where parents must complete redetermination in person. The BIAS study focused on 
Marion County because it includes the city of Indianapolis, which is home to approximately 35 percent of 
families receiving CCDF-subsidized child care in the state. 

The Office of Early Childhood and Out-of-School Learning (OECOSL) recognized that many parents 
missed their scheduled redetermination appointments and had to make multiple visits to an office to 
complete the redetermination process because they did not have all the needed documents. Eligible 
families in this situation risked missing the redetermination deadline and having to appeal or reapply 
a short time later (a phenomenon called “churning”), 2 or going without a subsidy for which they were 
eligible. Many states and public benefits programs face the same challenges with the child care redeter- 
mination process. 3 The BIAS study discussed in this chapter, which involved two rounds of tests, aimed 
to increase attendance at scheduled appointments and reduce the number of appointments parents had 
to attend to renew their child care subsidies. Improvements in these areas could have also reduced the 
number of eligible parents who missed the redetermination deadline. 

Define 

Parents in Marion County had to prove they were still eligible by providing documentation and attending 
an appointment at an intake office at least every six months. The OECOSL was interested in improving 
the redetermination process for parents enrolled in the CCDF program. The office identified several inter- 
related issues, which create burdens for both parents and staff: 

• Some parents missed their scheduled appointment, giving them less time to complete the 
redetermination process by the deadline and creating congestion in the office at the end of 
the redetermination period. 

• Some parents were not completing the redetermination process in one appointment, and mul- 
tiple visits to the office require more time and energy for parents and staff. 

• Some parents were not completing redetermination by the deadline, leading to a potential 
loss of benefits and gaps in service. 


1 The Child Care and Development Block Grant Act of 2014 requires states to extend re determination periods to 12 months. 

2 The state's appeal process allows parents who have recently lost their child care subsidies to appeal immediately and have 
them reinstated with retroactive pay to the provider if they can prove uninterrupted eligibility. This procedure reduces the 
number of parents who must reapply to the CCDF program and maintains continuity of care (and payments to the provider), 
but it also provides the administrators with some discretion about how to handle cases in which a parent continually fails to 
complete redetermination on time. 

3 See, for example, Rosenbaum (2015). 
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Diagnose — Round 1 

The Children’s Bureau, Inc. (CBI), is the CCDF intake and eligibility contractor in Marion County. It 
requires that every parent in the county attend an in-person appointment to complete redetermination. 4 
Subsidies in Indiana are usually valid for between eight weeks and six months, depending on the par- 
ent’s service need, but most parents have a semiannual redetermination deadline. 

As required by the state, CBI sends letters notifying parents about the expiration of their subsidies 
about 45 days before the deadline. The letter provides the date and time of the appointment, and informs 
parents that they may reschedule, if necessary. With the notification letter, CBI sends a checklist that 
explains acceptable forms of documentation. Parents are required to bring in a completed provider infor- 
mation form and proof of identity, address, and service need. 

The team collected data from the OECOSL and CBI about the redetermination process, and spoke 
to frontline and supervisory staff at the intake office, the regional OECOSL manager of the intake offices, 
and some parents who had just attended a redetermination appointment. In interviews with the BIAS 
team, CBI staff estimated that about 40 percent of parents missed their scheduled redetermination 
appointment. Data maintained by CBI showed that, in a typical month, about 35 percent of parents who 
attended an appointment had to schedule another appointment because of missing documentation. 5 
OECOSL data indicated that about 17 percent of parents did not renew their subsidies each month, and 
the OECOSL believed a proportion of these parents were still eligible for the benefit since the office regu- 
larly received appeals from parents who had missed the redetermination deadline. There were no reliable 
statistics on the proportion of parents who filed appeals or how many had their subsidies reinstated. 

During the diagnosis phase, the BIAS team identified two contextual factors, which ultimately the 
intervention could not address, and several bottlenecks that the team believed were hindering parents 
most from successfully completing the redetermination process. The contextual factors, identified bottle- 
necks, and their associated behavioral terms are described below (and displayed in Figure 3.1). Full defi- 
nitions of these terms, which appear in boldface the first time they appear in this chapter, are presented 
in Appendix Table A.l. 

The first contextual factor is the requirement that parents must visit the intake office to complete 
redetermination at least every six months, rather than sending forms electronically or by mail. This 
creates hassle factors that can be barriers to completing the process successfully. The second factor 
relates to the redetermination requirements themselves, all of which are determined by policy or statute. 
It can be difficult to comply with the requirements — particularly those related to employment verifica- 
tion. Addressing these barriers falls outside the scope of the BIAS project, but the new Child Care and 
Development Block Grant reauthorization points to the importance of addressing both of these problems 
in the future. 

Bottleneck 1: Parents may not receive or read the letter. 

CBI has no consistent policies for contacting parents when their letters are returned to the office and 
ensuring they receive the information about their appointments. Staff sometimes call parents when a 
phone number is available, but no other action is taken. For parents who do not receive the letter, staff 
assume the parents would remember to keep track of when they were up for renewal and contact the 
office, which places a heavy cognitive load on parents. Some parents who receive the letter but have 
been through the process before may only skim the materials for their appointment date because they 
already know what they need to complete the process (overconfidence effect). Though these parents 
likely understand the process, their circumstances may have changed between redetermination periods 
(for example, a change in work hours or jobs) in ways that require additional documentation. 


4 In some counties, re determination is completed by mail. The local contractor decides which process to use based on a variety 
of considerations. 

5 An additional 11 percent had to return the same day to drop off additional paperwork. In the outcome data reported for the 
Round 1 and Round 2 redetermination tests, dropping off additional paperwork is not counted as an additional appointment 
because parents do not typically see a case worker again when they are dropping off documentation. 
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FIGURE 3.1 

BEHAVIORAL MAP FOR RENEWING A CCDF SUBSIDY ON TIME 

INDIANA OFFICE OF EARLY CHILDHOOD AND OUT-OF-SCHOOL LEARNING 


CCDF REDETERMINATION PROCESS PROCESS BOTTLENECK HYPOTHESIZED BEHAVIORAL REASONS 

FOR THE BOTTLENECK 



SOURCE: Figure based on BIAS behavioral diagnosis research with staff from the Indiana Office of Early Childhood and Out-of-School Learning. 

NOTE: This map is a stylized representation of the child care redetermination process for some parents. A parent does not necessarily go through 
all these steps or move through them in the displayed order. 


Bottleneck 2: Collecting the right documentation to complete redetermination is challenging 
for parents. 


The letter parents receive about the documents they must bring to their appointment is dense and de- 
tailed, which may cause high deliberation costs to understand. The requirements to complete redeter- 
mination can be difficult for parents in the following ways: 


• Proof of service need, especially employment, can be hard to establish because the verifying 
documents must include specific information and dates. 


Parents must provide evidence of work or school to demonstrate a service need. To verify qualifying 
employment, parents must present proof of “30 days of income prior to your appointment,” typically 
in the form of pay stubs, copies of checks, or documentation from the employer if the job is new. To 
do so, parents may have to wait for a pay stub that has not been issued at the time of their appoint- 
ment, although staff attempt to implement the policy in a way that avoids unreasonable results. The 
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documents must also show work hours or an hourly rate for the last 30 days, which are not always 
included in pay stubs. Those who work in more unconventional or “personal service” jobs and are 
paid by deposit onto a stored value card (for example, a Visa debit card) or by personal check may 
have even more difficulty identifying the right materials. Lastly, parents must be organized and start 
setting aside pay stubs the day they receive their redetermination notice. Taken together, these has- 
sle factors place a heavy cognitive load on low-income parents — in this case, mostly single mothers 
who are working or in school. Forgetting to obtain a document (prospective memory failure) or 
procrastinating may lead to multiple redetermination appointments. 

• The provider information form must be filled out each time the parent completes the redeter- 
mination process. 

Parents may assume that the form is not necessary unless they are changing providers. There are 
also hassle factors in completing the form because it requires the provider’s information and signa- 
ture. Parents may intend to bring the form when they go to their child care provider but then forget it 
(prospective memory failure). Furthermore, only certain provider staff members can fill out the form, 
and thus parents who procrastinate may not meet with the right person in time for the appointment. 

• Parents may assume that showing proof of address is unnecessary or may not have a valid 
document at the time of their appointment. 

The checklist indicates that parents must bring in “proof of residency,” which means proof of 
address. The requirement is specific in that the proof must be “received within 30 days of the 
appointment.” A parent may misunderstand the requirement and bring in an out-of-date document. 
The checklist provides examples of qualifying documents, but if parents do not have one of those 
documents, they may struggle to determine what they should bring in as proof. The address on 
parents’ drivers licenses may not be current, or they may be living with someone else and not have a 
lease or utility bill in their name. 


Bottleneck 3: Parents may not remember or appropriately plan for their appointment. 

Parents who qualify for the CCDF subsidy are usually working or in school. 6 Given their busy schedules, 
the likelihood that they are managing other public assistance redetermination procedures, and the anxi- 
ety created by the imminent deadline for renewing the subsidy, parents are likely carrying a heavy cog- 
nitive load. Parents must set aside substantial time for the appointment and begin to gather the required 
documentation right away. All of these tasks can lead to prospective memory failure. The OECOSL sends 
the notice approximately four weeks before the scheduled appointment. Parents may simply forget about 
the appointment, or optimistically assume they will be prepared by that date, but decide not to go at the 
last minute because they do not have all the documents. 

Design — Round 1 

The BIAS team encouraged OECOSL staff to consider eliminating some of the requirements or shifting 
some of the burden to the employer or provider. For example, the BIAS team suggested that the rede- 
termination notification letter or the postmarked envelope it came in could serve as proof of residency, 
as a way to simplify the requirement. Since OECOSL staff could not know with certainty that a parent 
received the letter “within 30 days of the appointment,” they determined that the letter could not be 
used as proof. Given the limited time frame of the BIAS studies, the research team determined that the 
redetermination requirements could not be changed, and the tests therefore focused on attempting to 
make them clearer. 


6 The work requirement does not apply to parents whose children are separately eligible because these children already receive, 
or need to receive, protective services. 
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TABLE 3.1 

HYPOTHESIZED RELATIONSHIPS OF BOTTLENECKS, BEHAVIORAL CONCEPTS, 
AND COMPONENTS OF THE INTERVENTION - REDETERMINATION TEST 

INDIANA OFFICE OF EARLY CHILDHOOD AND OUT-OF-SCHOOL LEARNING 


INTERVENTION COMPONENTS 3 


HYPOTHESIZED BOTTLENECK AND 

BEHAVIORAL CONCEPTS 

Simplification 

Plan Making 

Reminders 

Choice 

Architecture 

PARENTS MAY NOT RECEIVE OR READ LETTER 

- 




Cognitive load 





Overconfidence effect 






COLLECTING THE RIGHT DOCUMENTATION TO COMPLETE REDETERMINATION IS CHALLENGING FOR PARENTS. 


Deliberation costs 





Hassle factors 





Cognitive load 





Prospective memory 





Procrastination 






PARENTS MAY NOT REMEMBER OR APPROPRIATELY PLAN FOR THEIR APPOINTMENT. 


Cognitive load 


Prospective memory 



NOTES: Behavioral concepts cannot be definitively identified, but rather are hypotheses derived from the behavioral diagnosis and design 
process that may explain behavioral bottlenecks. 

a The following are examples of intervention components in Indiana: 

Simplification: Provide fewer proof of residency options. 

Plan making: Reorganize the checklist to make it more user-friendly. 

Reminders: Send a reminder postcard. 

Choice architecture: Provide more detailed instructions for documenting proof of work. 


The BIAS team redesigned the notification letter and created a new reminder postcard. These 
materials aimed to help parents remember their appointments and understand the requirements. The 
intervention materials contained the behavioral elements of simplification, personalization, loss 
aversion, and reminders, described below (Table 3.1 connects hypothesized bottlenecks, behavioral 
concepts, and intervention components): 

• The redesigned letter excluded unnecessary content, presenting only the key information 
that parents needed to remember. It began with the parent’s name, which was handwritten to 
personalize the interaction. The letter prominently displayed the subsidy expiration date and 
stated, “Start to prepare TODAY!” to elicit a sense of urgency. It also activated loss aversion in 
parents with the phrase, “to avoid losing your CCDF, you need to.” 

• The streamlined checklist served as a plan-making device as well. It began with a clear 
title, “4 Easy Steps to Renewing Your Voucher,” to focus attention on the four essential pieces 
of documentation parents must bring to their appointment: proof of current address, proof 

of identity, the provider information form, and proof of eligibility (service need). It reduced 
the amount of text, compared with the standard checklist, and eliminated any unnecessary 
information, such as acronyms, that parents were unlikely to understand. It also included 
questions to help guide parents in determining what they needed to provide. The reverse side 
of the checklist also featured “Helpful Hints,” which included “Frequently Asked Questions” 
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about the redetermination process. Figure 3.2 contains the redesigned letter and checklist 
used in Round 1. 

• The provider information form was printed on yellow paper to make it stand out from 
other pieces in the packet and help remind parents to fill it out. 7 

• A reminder postcard, similar to a doctor appointment reminder, was mailed to parents 
about one to two weeks before the appointment. Parents had not been previously sent a re- 
minder. The postcard did not contain the specific date and time of the parent’s appointment, 
but encouraged them to consult the notification letter and write down the information. The 
postcard reminded parents about the four pieces of documentation they needed to bring to 
the appointment, referencing the checklist, and about what to do if they had any questions or 
were unable to attend. The postcard itself could serve as proof of address, and it encouraged 
parents to bring it to the appointment. 

Test — Round 1 

The test ran from January to June 2014, and the BIAS team evaluated it using a random assignment 
design. The team measured impacts on three main outcomes: 

• Percentage of parents attending their first scheduled appointment 

• Percentage of parents completing the redetermination process in one appointment 8 

• Percentage of parents completing the redetermination process on time 

The research team divided the sample equally between an intervention group and a control group. A 
research intern mailed all of the intervention materials to avoid disrupting normal business operations. 
Office managers set aside the notification letters for parents assigned to the intervention group, and the 
intern created the mailing packets. The intern then mailed the notification letters to both the control and 
intervention groups at the same time. The intern also sent the intervention group reminder postcards 
close to the date of the appointment. 

Findings — Round 1 

Implementation 

• The BIAS team modified the notification letter slightly during the test in response to 
feedback from CBI staff. 

The BIAS team received feedback from CBI managers during the intervention that some parents in 
the intervention group were confused about the date of their redetermination appointment. The stan- 
dard notification letter, which parents who had renewed their subsidies before had received, listed the 
appointment date at the top right corner. The redesigned letter noted the subsidy expiration date at the 
top and the appointment date further down the page. A few parents had arrived at the office very close 
to the day their subsidy expired, mistaking the date on the top of the redesigned notification letter for 
their appointment date. It was not possible to determine how many parents experienced this confusion. 
To clarify the information, the team made two changes to the redesigned letter that took effect on April 
14, 2014. The text in the center box changed from “CCDF Voucher Expiration Date” to “Your Voucher 
Expires On,” and the text in the right-hand box changed from “When To Renew” to “Your Appointment 
Date.” The team compared impacts for parents in the intervention group who received the original ver- 
sion of the letter with those who received the modified version and found no statistically significant 


7 The BIAS team considered redesigning this form, but decided against it because providers were responsible for completing it 
and changing the form for some but not all parents might cause confusion. 

8 Data on whether or not parents renewed after the deadline were not available, so this analysis reported on the number of 
appointments parents needed to renew only for parents who renewed on time. 
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FIGURE 3.2 

REDESIGNED REDETERMINATION MATERIALS EXCERPT, ROUND 1 

INDIANA OFFICE OF EARLY CHILDHOOD AND OUT-OF-SCHOOL LEARNING 


NOTIFICATION LETTER 


;ichildrenb bureau, Inc. 

^ \ Preserving families and protecting the future of Indiana’s children. 


Dear Jcvn&Voe/ 


Handwritten, personalized 
information 


Your CCDF voucher provides you with affordable child care that meets your 
needs. But your voucher will expire soon. 


Activate loss aversion 


Your Voucher Expires On: 

5 / 24 7 2014 


Elicit urgency 


Month Date 


Year 


To avoid losing your CCDF, you need to: 


D Gather just a few documents to show that 
you are still eligible for the voucher. 

WHAT TO BRING? 
> 

Details on the next page 


B Bring these documents to an appointment. 

YOUR APPOINTMENT 


DATE: 

If you need to reschedule, 

5/22/14 

call the Children's Bureau Inc. 

at 1-866-287-2420 x204 

or email us at 

At 9:30 AM 

CCDFscheduling@childrensbureau.org 

WHERE? 


Fay Biccard Glick 

Start to prepare TODAY! 

Family Place 

3801 N. Temple Ave. 
Indianapolis, IN 46205 


(continued) 
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FIGURE 3.2 

(CONTINUED) 


4 EASY STEPS CHECKLIST 


4 Easy Steps to Renewing Your Voucher 

Here is a checklist to help you tr^ck what you will need to bring to the appointment. 

I ) 1. Proof of Current Address 

Check off the onedocument you will bring: Streamlined checklist 

O Driver's license O Utility bill- current at time of appointment 

O Envelope postmarked in the 30 days prior to your appointment (no window envelopes) 

O Letter from State or Federal Government agency dated in the 30 days prior to your appointment 

2. Proof of Identity 

Check off the one document you ■/"'// bring: 

O Driver's license G M ;l 'tary ID 

O Passport O School ID Simplified information 

and text 

O Other government-issued ID O Work ID 

□ 3. Provider Information Page 

This is the attached YELLOW form. You must have your CCDF eligible provider fill it out 
even if you are keeping the same provider. 


□ 


4. Proof of Eligibility 

Check off all of the situations that apply to you and what you need to bring. 

1 1 Do you go to school or training? 

BRING A SCHEDULE that shows your name, school name, credit hours taken and/or hours of 
participation, and semester dates or begin and end dates, if applicable. 

f 1 Do you work? 

BRING PAY STUBS for all of your jobs for the 30 days prior to your appointment. 

Examples: If your appointment was on December 25, your pay stubs would have to cover the days from November 
25 to December 25. If you had two jobs that paid every other week, you would have to bring four pay stubs to your 
appointment, two for each job. 

I I Do you have any other forms of income, aside from work? 

BRING PROOF OF ANY OTHER INCOME, for example: 

O Unemployment printout 
O Social Security (SSI) benefit letter 
O Current TANF benefit letter 


Questions about what to bring? 
See the next page. 


Need more help? 

Call us at 1-866-287-2420 x204 
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TABLE 3.2 

REDETERMINATION OUTCOMES, ROUND 1 

INDIANA OFFICE OF EARLY CHILDHOOD AND OUT-OF-SCHOOL LEARNING 


OUTCOME 

PROGRAM 

GROUP 

CONTROL 

GROUP 

DIFFERENCE 

Attendance 

Attended first scheduled appointment (%) 

52.6 

50.0 

2.6* 

Number of appointments attended (%) 

None 

9.2 

9.6 

-0.5 

One 

71.1 

69.1 

2.1* 

More than one 

19.7 

21.3 

-1.6 

Redetermination 

Completed redetermination on time (%) 

79.3 

77.7 

1.6 

Completed in one appointment 

62.5 

59.3 

3.2** 

Completed in more than one appointment 

16.7 

18.4 

-1.7 

Sample size (total = 5,332) 

2,666 

2,666 



SOURCE: MDRC calculations based on data from the Office of Early Childhood and Out-of-School Learning. 

NOTES: Estimates were regression-adjusted using ordinary least squares, controlling for pre-random assign- 
ment characteristics of sample members. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 
Rounding may cause slight discrepancies in sums and differences. 


differences in impacts between the two groups. In other words, the intervention materials were no more 
effective after April 14, 2014, than they were before that date. 

Impacts 

• The intervention increased the percentage of parents who attended their first scheduled 
redetermination appointment by 2.6 percentage points, a 5 percent change. 

Table 3.2 shows that 50.0 percent of control group members attended their first scheduled appoint- 
ment, compared with 52.6 percent of the intervention group. The difference is statistically significant. 

• The intervention increased the percentage of parents who completed redetermination in 
one appointment by 3.2 percentage points, or about 5 percent. 

Table 3.2 shows that 62.5 percent of parents in the intervention group were able to renew their sub- 
sidies in one appointment, compared with 59.3 percent of control group parents. This impact reduces the 
burden associated with the redetermination process on both parents and CBI staff. 

• The intervention did not change the likelihood that parents would renew their subsidies 
on time. 

The improvement in the proportion of parents who completed the redetermination process in one 
appointment did not translate into a statistically significant effect on the proportion of parents who 
renewed their subsidies on time because enough parents in the control group caught up to those in the 
intervention group by attending multiple appointments. About 18.4 percent of the control group (and 16.7 
percent of the intervention group) attended two or more appointments to complete the process before the 
deadline. 
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• The intervention’s impact on completing the redetermination process in one appoint- 
ment seems to result from a general improvement in parents’ ability to comply with the 
redetermination requirements. 

Appendix Table A. 2 shows the reasons why parents did not renew in a given appointment, all of 
which the intervention materials attempted to address. A smaller percentage of parents in the interven- 
tion group, compared with those in the control group, did not complete the redetermination process in an 
appointment for one or more of the listed reasons (except proof of identity), suggesting that the interven- 
tion generally improved parents’ understanding of the redetermination requirements. However, these 
differences were not large enough to indicate statistically significant effects. Problems with employment 
verification documents accounted for the primary reason why parents did not complete the process in 
any appointment, a finding that is consistent with past research in this area. 9 

Subgroup Analysis 

The subgroup analysis considered whether impacts varied based on the office to which a parent was 
assigned and on the parent’s educational level. The BIAS team chose these exploratory subgroups be- 
cause different offices might have different practices, cultures, or workloads, which could cause impacts 
to vary, and a parent’s educational level might affect how much help the person needs to complete the 
redetermination process. For example, all of the notification materials are provided in writing, requiring 
at least a basic reading level even for the intervention group. 10 

• The behavioral intervention’s effects differed based on the office to which the parent 
was assigned and on the parent’s level of education. 

The intervention more positively affected parents assigned to the largest CBI office with the most 
customers (Family Place) than parents assigned to less busy offices. Table 3.3 shows that 61.6 percent of 
parents who received the BIAS materials at Family Place renewed in one appointment, compared with 
55.8 percent of the control group parents at that location (a difference of 5.8 percentage points, or 10 per- 
cent). For parents with lower educational levels (less than a high school degree), the intervention reduced 
the percentage who renewed through multiple visits to the office by 7.0 percentage points, or 33 percent, 
suggesting the materials were particularly effective for these parents at clarifying the requirements. 

At the end of Round 1, it seemed possible to further increase the percentage of parents who 
attended their first scheduled appointment. Although separate from completing the redetermination 
process in one appointment, this outcome is useful in encouraging parents to begin the process as early 
as possible. CBI offices become more crowded as the redetermination deadline nears, and parents have 
fewer chances to schedule additional appointments to resolve any problems with their documentation. 
The BIAS team had also hoped the redesigned notification materials would statistically significantly 
reduce the type and number of errors that lead to multiple redetermination appointments. Many human 
services programs attempt to explain complicated redetermination requirements to participants using 
written notices, so this outcome could have been an important achievement for the human services field. 
Thus, with the support of state partners, the BIAS team designed another intervention. 

Diagnose — Round 2 

The BIAS team conducted additional interviews with program staff, observed program operations, and 
spoke to a small number of parents who received the intervention and control materials. The team identi- 
fied the following persistent bottlenecks: 


9 Sandstrom, Grazi, and Henly (2015). 

10 Other exploratory subgroups analyzed include the number of children in the household, family income, the month the family 
was scheduled for redetermination, and the number of prior redetermination cycles. The research team chose these subgroups 
because the number of children and family income might influence a parent's need for child care assistance, a parent’s need or 
ability to complete the redetermination process might vary by season or month in relation to the weather or academic year, and 
families with less experience with the process might benefit more from the intervention materials. Results for these subgroups 
are not shown because there were no consistent patterns of statistically significant results. 
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TABLE 3.3 

REDETERMINATION OUTCOMES BY SUBGROUP, ROUND 1 

INDIANA OFFICE OF EARLY CHILDHOOD AND OUT-OF-SCHOOL LEARNING 


SUBGROUP AND OUTCOME 

PROGRAM 

GROUP 

CONTROL 

GROUP 

DIFFERENCE 

ACROSS- 

SUBGROUP 

SIGNIFICANCE 

Office 

Attended first scheduled appointment (%) 

Family Place 

52.7 

50.0 

2.7 


Family Support Center 

49.0 

49.2 

-0.2 


NACS West 

55.2 

50.6 

4.6* 


Completed redetermination on time (%) 

Family Place 

78.0 

75.5 

2.5 


Family Support Center 

79.2 

78.8 

0.3 


NACS West 

81.5 

80.2 

1.2 


Completed in one appointment (%) 




t 

Family Place 

61.6 

55.8 

5.8*** 


Family Support Center 

62.7 

64.4 

-1.7 


NACS West 

63.9 

60.6 

3.3 


Completed in more than one appointment (%) 

Family Place 

16.4 

19.7 

-3.3** 


Family Support Center 

16.4 

14.4 

2.0 


NACS West 

17.6 

19.7 

-2.1 


Education 

Attended first scheduled appointment (%) 

Not a high school graduate 

52.2 

52.6 

-0.4 


High school graduate or higher 

52.7 

49.3 

3.4** 


Completed redetermination on time (%) 

Not a high school graduate 

73.4 

74.4 

-0.9 


High school graduate or higher 

81.0 

78.7 

2.3* 


Completed in one appointment (%) 

Not a high school graduate 

58.8 

52.7 

6.1** 


High school graduate or higher 

63.6 

61.2 

2.4 


Completed in more than one appointment (%) 




ttt 

Not a high school graduate 

14.5 

21.5 

—7.0*** 


High school graduate or higher 

17.4 

17.5 

-0.1 



SOURCE: MDRC calculations based on data from the Office of Early Childhood and Out-of-School Learning. 

NOTES: Estimates were regression-adjusted using ordinary least squares, controlling for pre-random assignment characteristics 
of sample members. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Differences across subgroup impacts were tested for statistical significance. Those statistical significance levels are indicated 
as follows: ttt = 1 percent; tt = 5 percent; t = 10 percent. 

Rounding may cause slight discrepancies in differences. 

The sample sizes for the office subgroup are: Family Place = 2,487; Family Support Center = 1,283; NACS West = 1,562. The 
sample sizes for the education subgroup are: not a high school graduate = 1,155; high school graduate or higher = 4,167. 
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Bottleneck 1: Because redetermination is a frequent event, some parents only skim the letter 
for their appointment date and assume that the requirements for them have not changed. 

Parents who have renewed their subsidies before believe they know what is required to complete the 
process. However, changes in parents’ circumstances may require new documents, which could result 
in multiple appointments if they do not read the packet carefully. Other parents may find the process to 
be so complicated that they attend their first appointment to receive staff assistance, knowing full well 
they do not have all the documents and will have to return to complete the process. 

Bottleneck 2: Parents have considerable difficulty submitting proof of work and other forms 

of income. 11 Even though the Round 1 materials attempted to simplify the instructions, the employment 
verification documents may have remained complicated to many parents. According to data collected 
during Round 1, missing pay stubs and missing documents for other forms of income (such as unem- 
ployment insurance or Temporary Assistance for Needy Families) were the top reasons parents failed to 
renew their subsidies in a given appointment. 

Bottleneck 3: Parents miss their scheduled appointment. The postcard reminder in Round 1 did 
not include the parent’s appointment date and time, and therefore it was not helpful to parents who had 
forgotten their appointment date (prospective memory failure). It was not possible to automate the cre- 
ation of the postcards given the small scale of the intervention in Round 1. 

Design — Round 2 

In the second round of the experiment, the BIAS team set out again to reduce the number of parents 
who had to return to the office because of a problem with their work verification documents, the most 
common reason for additional appointments in Round 1 . The team reviewed a packet that CBI used in a 
different county, which offered redetermination by mail to parents. It provided a model of the information 
parents would need to complete the redetermination process without an office appointment. This packet 
contained several forms that the CCDF did not regularly mail to parents in Marion County. Although it 
increased the size of the packet, the BIAS team decided to include some of these forms in the Round 
2 intervention materials to test if they could reduce the need for follow-up visits and demonstrate the 
feasibility of offering redetermination by mail in Marion County. 12 The Round 2 intervention materials 
contained the following elements: 

• A revised notification letter to encourage parents to read all the information. 

Behavioral research has shown that people respond to visual cues and quick summaries of 
the information they need. 13 The BIAS team completely redesigned the appointment letter 
to resemble a vertical roadmap with the appointment date and time situated in the middle. 

The team chose this design to counteract the tendency for parents to scan the top of the 
page for the appointment time and disregard the rest of the text, which they often assume is 
unimportant. The top of the roadmap, above the appointment date, emphasized the materials 
parents would need to bring to the appointment, especially proof of employment. Below the 
appointment date and separated from it to avoid confusion, the letter noted the subsidy expi- 
ration date. As in Round 1, the letter began with a handwritten, personalized greeting. 

• More information about work and income requirements. The BIAS team created a 
new insert and added it after the checklist. The insert included a checklist outlining different 
scenarios in which a parent would have to bring in particular documents, a pay stubs work- 
sheet personalized with the exact date range for each parent to help clients determine what 


11 Sandstrom, Grazi, and Henly (2015); Johnson-Staub, Matthews, and Adams (2015). 

12 The BIAS team had encouraged the site to consider offering redetermination by mail or drop-off, at least for some parents (for 
example, those who had completed the redetermination process successfully several times before). 

13 Petersen et al. (2007); Wansink, Painter, and North (2005). 
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they needed to collect, and the two forms (self-employment and new job forms) that were not 
in the Round 1 intervention or control packets. Even with these guidance documents, it was 
difficult to fully explain the required work documentation. The BIAS team aimed to shift 
some of the burden of understanding the requirement away from parents and back to staff by 
encouraging parents to bring all recent pay stubs from all jobs that they received, with the 
intention that staff could sort through the paystubs to identify the date range needed at the 
appointment. 

• Personalized reminder postcards with the client’s appointment time and date. In 

Round 2, a CBI staff member wrote each parent’s appointment date and time on the reminder 
postcard. In comparison to Round 1, CBI sent the reminder slightly earlier (about one week 
after it mailed the initial appointment packet) to give parents more time to prepare and follow 
up. (Figure 3.3 presents the redesigned letter and reminder postcard for Round 2.) 

Test — Round 2 

The test ran from July to November 2014, and the BIAS team evaluated it using a random assignment 
design. The team divided the sample members equally between an intervention group and a control 
group. 14 Table 3.4 contrasts the materials that the intervention and control groups received in the two 
rounds. In Round 2, the BIAS team randomly assigned 1,599 individuals from the Round 1 intervention 
group whose redetermination deadlines came up again during this period. These parents represented 
about one-third of the Round 2 sample. 

Findings — Round 2 

Impacts 

• The intervention increased the percentage of parents who attended their first scheduled 
redetermination appointment by 10.6 percentage points, a 24 percent change. 

More parents in the intervention group started the redetermination process earlier by attending 
their first scheduled appointment. Table 3.5 shows that 54.7 percent of parents in the intervention group 
attended their first scheduled appointment, compared with 44.1 percent of those in the control group. 

• The intervention did not increase the percentage of parents who completed redetermi- 
nation in one appointment, and did not have an effect on the reasons parents did not 
complete redetermination in a given appointment. 

About the same percentage of parents in the intervention and control groups completed redetermi- 
nation in one appointment (58.0 percent in the intervention group and 56.5 percent in the control group). 
There were no statistically significant differences in the reasons why parents did not complete redeter- 
mination in a given appointment (Appendix Table A. 3). The work verification insert that was added to 
the Round 2 notification materials did not increase parents’ understanding of or ability to meet the work 
verification requirement. 

• The intervention increased the percentage of parents who renewed on time by 2.7 
percentage points, or 4 percent. 


14 About 20 percent of the families in the sample went through the redetermination process more than once during the study 
period and were assigned to their original group for subsequent assignments. This analysis reported on only the outcomes 
from the family’s initial redetermination application during the study period and not the subsequent ones. 
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FIGURE 3.3 

REDESIGNED REDETERMINATION MATERIALS EXCERPT, ROUND 2 

INDIANA OFFICE OF EARLY CHILDHOOD AND OUT-OF-SCHOOL LEARNING 


NOTIFICATION LETTER 


itchf/drerte bureau, Inc, 

^ \ Preserving families and protecting the future of Indiana's children 


Janes Voe^ 

It's our goal to get you reauthorized in one appointment so you don't have to keep coming back. 
Why? Because we know you'd rather focus on family, work, or school - not CCDF reauthorization. 
This newly designed packet is all about helping you get this right the first time. Save time later by 
making sure you have all of the right documents! 


Visual cues 




o 


Start to prepare today! 


Working? 

You need to show 
proof of work from 

7/26/14 to 

8/25/2014 

(based on your 
current 
appointment). 
Get help on Page 4! 


Pay close attention to anything that has changed since your last appointment. 
Read the following pages to make sure you bring the right documents. 

1) Proof of Current Address (page 2) 

2) Proof of Identity (page 2) 

3) Provider Information Page (page 9) 

4) Proof of Work (page 4) or School (page 2) 


Emphasize requirements 
first, then share 
appointment date 


0 


Attend your appointment on 


August 25, 2014- at lOanv 



Put a reminder in your 
phone of your 
appointment dote & time. 


0 


Fay Biccard Glick Family Place 
3801 N. Temple Ave. 

Indianapolis, IN 46205 

Need to reschedule or have questions? 

Call us NOW at 1-866-287-2420 x 204 
or email CCDFscheduling@childrensbureau.org 

We will respond within one business day. 


Recertify before your voucher expires. 

Your voucher expiration date is September 6, 2014. 


Page 1 


Turn the page! 


(continued) 
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FIGURE 3.3 

(CONTINUED) 


Personalized information 


REMINDER POSTCARD 


Your CCDF 
appointment is on: 

August 25, 2014 at 10am/ 

Unless you rescheduled recently 

Fay Biccard Glick Family Place 
3801 N. Temple Ave. 

Remember to bring: 

□ Proof of Address 

The postmarked envelope this 
postcard came in will work. 

□ Proof of ID 

For example: a driver’s license 

O Provider Information Page 

Filled out by your provider 

□ Proof of Work or School 

Tip: bring more pay stubs than you 
think you need! 

□ Proof of Other Income 

For example: Ul, SSI, TANF letter 

D Your swipe card 


Questions? 

Call 1-866-287-2420 x204 - remember to 
leave a name and number -or email 

CCDFscheduling@childrensbureau.org 
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TABLE 3.4 

OVERVIEW OF PROGRAM AND CONTROL GROUP MATERIALS, 
REDETERMINATION TESTS 

INDIANA OFFICE OF EARLY CHILDHOOD AND OUT-OF-SCHOOL LEARNING 


MATERIALS 

CONTROL 
GROUP, BOTH 
ROUNDS 

PROGRAM GROUP, 
ROUND 1 

PROGRAM GROUP, 
ROUND 2 

Notification letter 

✓ 

Redesigned 

Redesigned 

Checklist 

✓ 

Redesigned 

Updated from 

Round 1 version 

Helpful hints page 


New 

Updated from 

Round 1 version 

Rules and responsibilities page 

✓ 



Proof of work insert 



New 

Provider information form 

✓ 

Printed on yellow paper 

Updated from 

Round 1 version 

Reminder postcard 


New 

New (personalized) 


TABLE 3.5 

REDETERMINATION OUTCOMES, ROUND 2 

INDIANA OFFICE OF EARLY CHILDHOOD AND OUT-OF-SCHOOL LEARNING 


OUTCOME 

PROGRAM 

GROUP 

CONTROL 

GROUP 

DIFFERENCE 

Attendance 

Attended first scheduled appointment (%) 

54.7 

44.1 

10.6*** 

Number of appointments attended (%) 

None 

14.4 

17.8 

-3.4*** 

One 

61.7 

60.3 

1.5 

More than one 

23.9 

21.9 

2.0 

Redetermination 

Completed redetermination on time (%) 

79.1 

76.4 

2.7** 

Completed in one appointment 

58.0 

56.5 

1.5 

Completed in more than one appointment 

21.1 

19.9 

1.2 

Sample size (total = 4,732) 

2,365 

2,367 



SOURCE: MDRC calculations based on data from the Office of Early Childhood and Out-of-School Learning. 

NOTES: Estimates were regression-adjusted using ordinary least squares, controlling for pre-random assign- 
ment characteristics of sample members. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 
Rounding may cause slight discrepancies in sums and differences. 
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TABLE 3.6 

REDETERMINATION OUTCOMES BY SUBGROUP, ROUND 2 

INDIANA OFFICE OF EARLY CHILDHOOD AND OUT-OF-SCHOOL LEARNING 


SUBGROUP AND OUTCOME 

PROGRAM 

GROUP 

CONTROL 

GROUP 

DIFFERENCE 

ACROSS- 

SUBGROUP 

SIGNIFICANCE 

Office 

Attended first scheduled appointment (%) 

Family Place 

52.7 

42.9 

9.9*** 


Family Support Center 

56.2 

46.4 

g <j *** 


NACS West 

56.9 

44.2 

12.8*** 


Completed redetermination on time (%) 




ttt 

Family Place 

77.3 

76.4 

0.8 


Family Support Center 

81.6 

71.3 

10.3*** 


NACS West 

80.1 

80.0 

0.2 


Completed in one appointment (%) 

Family Place 

55.6 

53.2 

2.4 


Family Support Center 

61.1 

56.9 

4.2 


NACS West 

59.7 

61.0 

-1.3 


Completed in more than one appointment (%) 




ft 

Family Place 

21.7 

23.2 

-1.6 


Family Support Center 

20.6 

14.4 

6.1*** 


NACS West 

20.5 

19.0 

1.5 



(continued) 


Table 3.5 shows that 79.1 percent of parents who received the intervention materials renewed on 
time, compared with 76.4 percent of parents in the control group. Renewing on time was not contingent 
on completing the redetermination process in one appointment since, as noted above, the intervention 
did not change the number of office visits parents had to make. 

• The intervention decreased the percentage of parents who did not attend any 
redetermination appointments by 3.4 percentage points, or 19 percent. 

Table 3.5 shows that 17.8 percent of parents in the control group did not attend any redetermination 
appointments, compared with 14.4 percent of parents in the intervention group. This result means that 
the BIAS materials encouraged 81 parents who would not otherwise have attended any redetermination 
appointments to attend at least one. 

Subgroup Analyses 

Similar to Round 1, the subgroup analysis examined whether impacts varied based on the office to 
which the parent was assigned and the parent’s educational level. In addition, the research team con- 
ducted an exploratory subgroup analysis to account for the fairly large number of parents from Round 1 
who were randomly assigned again in Round 2. These parents did not experience a different pattern of 
effects compared with other parents (data not shown). 15 

• The intervention increased on-time completion of the redetermination process at the 
smallest CCDF office, but more parents at this office had to attend more than one ap- 
pointment to complete redetermination. 


15 In addition, the same set of exploratory subgroups for Round 1 were analyzed for Round 2, but are not shown because there 
were no consistent patterns of statistically significant results. Those subgroups were the number of children in the household, 
family income, the month the family was scheduled for redetermination, and the number of prior redetermination cycles. 
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TABLE 3.6 

(CONTINUED) 


SUBGROUP AND OUTCOME 

PROGRAM 

GROUP 

CONTROL 

GROUP 

ACROSS- 

SUBGROUP 

DIFFERENCE SIGNIFICANCE 

Education 

Attended first scheduled appointment (%) 

Not a high school graduate 

58.3 

45.1 

13.2*** 

High school graduate or higher 

53.7 

43.9 

9.8*** 

Completed redetermination on time (%) 

Not a high school graduate 

76.7 

75.3 

1.5 

High school graduate or higher 

79.8 

76.8 

3.0** 

Completed in one appointment (%) 

Not a high school graduate 

55.4 

53.8 

1.7 

High school graduate or higher 

58.7 

57.3 

1.4 

Completed in more than one appointment (%) 

Not a high school graduate 

21.3 

21.5 

-0.2 

High school graduate or higher 

21.0 

19.4 

1.6 


SOURCE: MDRC calculations based on data from the Office of Early Childhood and Out-of-School Learning. 

NOTES: Estimates were regression-adjusted using ordinary least squares, controlling for pre-random assignment characteristics 
of sample members. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Differences across subgroup impacts were tested for statistical significance. Those statistical significance levels are indicated 
as follows: ttt = 1 percent; ft = 5 percent; t = 10 percent. 

Rounding may cause slight discrepancies in sums and differences. 

The sample sizes for the office subgroup are: Family Place = 2,249; Family Support Center = 1,092; NACS West = 1,391. The 
sample sizes for the education subgroup are: not a high school graduate = 1,041; high school graduate or higher = 3,688. 


Table 3.6 shows that the Round 2 intervention was more effective in the smallest office (Family Sup- 
port Center). Among parents served by this location, 81.6 percent of those who were sent the intervention 
materials completed redetermination on time, compared with 71.3 percent of the control group, which 
is a difference of 10.3 percentage points or about 14 percent. At the same time, the intervention in this 
office had a greater impact on increasing the number of parents renewing in multiple appointments. The 
reasons for this pattern were unclear. There were no statistically significant variations in the effective- 
ness of the intervention based on the parent’s level of education. 

Discussion 

The interventions in the two rounds seemed to have operated through different mechanisms, which may 
explain their different effects. The main achievement of the BIAS outreach in Round 1 was that it in- 
creased the percentage of parents who renewed their subsidies in one appointment. Because the reduc- 
tion in the number of office visits was greatest for parents who did not have a high school degree, it is 
likely that these materials effectively simplified the description of the paperwork requirements, making 
them more understandable for people with lower levels of education. The Round 1 intervention improved 
the efficiency of the process for some parents and CBI staff, but it did not increase the percentage of par- 
ents who completed the redetermination process on time by a statistically significant margin. 

In Round 2, the intervention increased the pipeline of parents who participated in CCDF redetermi- 
nation appointments, and got some parents started earlier. The BIAS materials encouraged a number 
of parents who would have otherwise not attended any appointments to attend at least one, and signifi- 
cantly increased the percentage who attended their first scheduled appointment, but there is no evi- 
dence that parents in this round had an improved understanding of the verification requirements. There 
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was no reduction in the number of parents who had to attend more than one appointment to complete 
redetermination, which means that the materials the BIAS team added to the notification packet in an 
effort to avoid additional appointments did not succeed. However, the intervention did have an impact 
on completing the redetermination process on time. This pattern of effects suggests that there may be a 
tension between inclusion and efficiency — parents who would otherwise not attend any appointments 
may have more challenging life circumstances or lower executive functioning skills. For them, the goal 
of renewing their subsidies in one appointment may be less important than initiating the process by 
attending an appointment and thereby improving their chances of keeping their subsidies. 

These findings suggest that elements of the intervention materials from both rounds could be com- 
bined to optimize impacts. The simple checklist of verification requirements might reduce the number 
of appointments for people who intended to renew their subsidies, and strong reminders such as the 
personalized postcard might encourage parents who would not otherwise do so to participate in the 
process. These tests also corroborate findings from past research about parents cycling off subsidies due 
to redetermination requirements, likely reflecting the challenges of communicating complex verification 
requirements to parents through the mail. 16 Employment verification is clearly the most difficult require- 
ment. This difficulty is due to the policy itself, which states that parents must show that they are work- 
ing during the 30 days before their appointment. This requirement puts parents in a difficult position of 
trying to coordinate their pay stub schedule with their appointment date. They may have irregular, shift- 
based work schedules, common among low- wage workers. The requirement can also cause confusion 
among parents about what documents to collect. The BIAS team tested two behavioral communications 
to improve parents’ ability to comply with this requirement, and neither changed the percentage of par- 
ents who struggled with it. While more behavioral outreach strategies can and should be tested, it is also 
worth considering whether the burden for satisfying this requirement could be shifted from the parent to 
the employer, as is done in the child support system, for example, in which employers submit payments 
directly to the state. It is probably easier for employers to produce the necessary information at the right 
time than it is for parents. Indiana has a website that employers could use to update income information 
for recipients of public benefits, but it is underused. Future behavioral work in this area should examine 
ways to expand its use. 


16 Forry, Daneri, and Howarth (2013). 
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CHAPTER 


Conclusion 


The previous chapters have provided a detailed analysis of the implementation and effects of the Paths 
to QUALITY (PTQ) child care selection and Child Care and Development Fund (CCDF) subsidy redeter- 
mination interventions in Indiana. In both cases, the BIAS tests modified existing communications and 
organizational resources to create behavioral interventions that could be implemented quickly and inex- 
pensively. Each intervention targeted individual parents and required only a few minutes of the person’s 
time to review the redesigned materials, participate in a phone call, or both. These interventions are 
examples of behavioral nudges that encourage people to make decisions that are in their best interest 
and do not eliminate options or substantially change incentives. These interventions were unobtrusive, 
and they succeeded in the following ways: 

• In the PTQ study, the referrals-phone intervention increased the use of highly rated providers 
(Level 3 or 4) by 2.1 percentage points, a 17 percent change. 

• In the first round of the redetermination study, the behavioral intervention produced the fol- 
lowing impacts: 

° An increase in the percentage of parents who attended the first scheduled appointment of 
2.6 percentage points, or 5 percent 

° An increase in the percentage of parents who completed the redetermination process in 
one appointment of 3.2 percentage points, or 5 percent 

• In the second round of the redetermination study, the behavioral intervention led to the follow- 
ing positive outcomes: 

• An increase in the percentage of parents who attended the first scheduled appointment of 
10.6 percentage points, or 24 percent 

° A reduction in the percentage of parents who did not attend any appointments of 3.4 per- 
centage points, or 19 percent 

° An increase in the percentage of parents who completed the redetermination process on 
time of 2.7 percentage points, or 4 percent 

These positive effects are meaningful. They demonstrate that changes in existing practices, 
informed by behavioral science, can improve outcomes. Yet, there were important outcomes that these 
communications-based behavioral interventions did not achieve. For example, the PTQ interventions 
did not increase the proportion of parents who chose quality-rated providers overall. The interventions 
did not significantly reduce the proportion of parents who had to come back to the office due to employ- 
ment verification issues. Future research in this area should therefore aim to strengthen messaging to 
individual parents, while also addressing bottlenecks in local child care systems and CCDF contexts. 
Behavioral interventions that are more intensive than the ones evaluated in this report would seek to 
change the supply of quality-rated child care providers and streamline administrative processes for large 
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groups of parents. These approaches recognize that there is a limit to returns on investments in better 
communications, given the pressures of everyday life for low-income working parents and the inherent 
complexity of child care decision making and the redetermination process. 

Two promising ideas for future studies include using behavioral techniques to recruit more providers 
to quality-rating and improvement systems (especially in underserved communities) and motivate them 
to improve their ratings over time, and getting employers more involved in the redetermination process. 
In line with the new Child Care and Development Block Grant Act’s call for states to create redetermina- 
tion policies that do not “unduly disrupt [parents’] employment,” future research may also investigate 
whether eliminating eligibility requirements would make the process less time consuming without 
jeopardizing the CCDF subsidy program’s integrity. The BIAS team’s diagnosis of the redetermination 
process brought attention to the bottlenecks related to parents’ understanding of and ability to comply 
with the verification requirements. This information and the data on the errors that led to multiple rede- 
termination appointments could be used to develop targeted reforms. For example, states should con- 
sider eliminating the provider information form, requiring providers to submit the information electroni- 
cally through the same system used to process payments, and allowing parents to use the appointment 
notification letter as proof of address. 

Even without these larger changes, the value of small nudges and the behavioral diagnosis and 
design process can be seen in Indiana and other states that participated in BIAS projects. The Indiana 
Association for Child Care Resource and Referral (IACCRR) proposed bringing the PTQ intervention to 
scale in Indiana. The association requested about $18,000 from the state for materials (paper, envelopes, 
postage, and so on) to provide referral packets to every new family who joins the waitlist each year. This 
investment translates into a per-family cost of about $1.40. In terms of labor, IACCRR estimated that 
creating and mailing referrals would require half the time of a full-time employee, and that conducting 
personal phone outreach would require two full-time consumer education staff members. IACCRR has 
closed its doors since this proposal was submitted, but the interest in bringing the intervention to scale 
and the cost estimates are informative. 

Behavioral economics provides a new way of thinking about the design of human services programs 
and a potentially powerful set of tools for improving program outcomes. The BIAS project offered the 
opportunity for continued hypothesis testing grounded in behavioral economics and takes advantage of 
the low-cost, iterative nature of rapid-cycle experimentation. In addition to work covered in previously 
published reports (see the list of previously published research at the back of this report), the BIAS proj- 
ect has forthcoming reports on an evaluation with the Washington State Division of Child Support and a 
synthesis of the project’s entire body of work. 
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APPENDIX TABLE A.l 
BEHAVIORAL TERMS 


TERM 

DEFINITION 

EXAMPLE TEXT 

Choice architecture 

The idea that decisions can be influenced by the way in which choices are pre- 
sented. For example, organ donation registration can be the default on license re- 
newals, requiring people to actually opt out if they do not want to be organ donors. 

Thaler and Sunstein 
(2008) 

Cognitive load 

Overburdened mental resources that impair individual decision making. People 
typically think that they will be able to pay attention to information and then 
understand and remember it as long as it is important. However, an individual’s 
mental resources — which are often taken for granted — are not unlimited and 
are more fallible than people often recognize. Challenges and emotional stress can 
drain these mental resources and actually make it difficult to make good decisions. 

Paas and van 

Merrienboer (1994) 

Deliberation costs 

The costs — in time or mental effort — of making a decision. 

Pringle (2006) 

Hassle factor 

A feature or situational detail that makes a behavior harder to adopt. A hassle 
factor could be, for example, a small barrier to completing a task, such as fill- 
ing out a form or waiting in line. While these factors may seem trivial and are 
often neglected in program design, reducing or eliminating them can have an 
outsized impact on outcomes. 

Bertrand, 

Mullainathan, and 
Shaflr (2004) 

Implementation 

prompt 

A self-regulatory strategy, sometimes referred to as an “if-then plan,” that in- 
creases the attainment of desired goals. The strategy takes the form of “when 
situation X arises, I will implement response Y.” An implementation prompt 
assists people in plan making, or forming implementation intentions, which 
can facilitate the fulfillment of goals. 

Milkman et al. (2011) 

Information 

overload 

Exposure to or provision of too much information or data. 

Toffler (1970) 

Loss aversion 

The tendency for decisions and behavior to be influenced by the desire to avoid 
a loss. When a decision is framed in terms of a loss or a gain, it affects the deci- 
sion maker's response. When loss aversion is operating, people experience a loss 
as more painful than when they experience a comparable gain as pleasurable. 

Kahneman, Knetsch, 
and Thaler (1990) 

Overconfidence 

The tendency for individuals to have excessive confidence in their ability to 

Moore and Healy 

effect 

overcome challenges, relative to their actual performance. 

(2008) 

Personalization 

Techniques designed to make communication less generic. 

Service et al. (2014) 

Present bias 

Giving more weight to present concerns than to future ones. People tend 
to make plans to do unpleasant tasks “tomorrow,” and they make the same 
choice when “tomorrow” becomes “today.” 

Laibson (1997) 

Primacy bias 

When individuals are biased toward selecting the first item on a list. 

Feenberg, Ganguli, 
Gaule, and Gruber 
(2015) 

Procrastination 

When people voluntarily and irrationally delay intended actions, despite the 
expectation of potentially negative consequences. 

Sirois and Pychyl 
(2013) 

Prospective 

memory 

Remembering to perform a planned action or intention at the appropriate time. 

Brandimonte, 

Einstein, and 
McDaniel (1996) 

Psychological 

The distance (spatial, temporal, or probable) between an individual and some 

Trope and Liberman 

distance 

outcome or decision. When an event is psychologically distant, it is perceived 
in an abstract manner and potentially important details are disregarded. 

(2010) 

Reference point 

Point of comparison, such as a past experience or a small contextual feature, 
that determines or influences people's subsequent reactions. That is, human 
beings' emotional responses to what happens to them are determined by, not 
the outcome itself in absolute terms, but the outcome relative to one’s refer- 
ence point. 

Kahneman and 
Tversky (1979) 


(continued) 
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APPENDIX TABLE A.l 

(CONTINUED) 


TERM 

DEFINITION 

EXAMPLE TEXT 

Reminder 

Prompting a specific piece of information to make it noticeable to an individ- 
ual and increase the chances of acting on that information. Reminders often 
work when they are related to something the individual intends to do. 

Karlan, McConnell, 
Mullainathan, and 
Zinman (2016) 

Simplification 

To reduce in complexity or make easier to understand. 

Service et al. (2014) 

Social norm 

Behavior that is established by others as a cue for one's own behavior, even when it 
is not directly relevant to a particular situation or person. For instance, people tend 
to perceive an outcome as more valuable if they see other people trying to attain 
that outcome. This psychological concept suggests that what matters is not just 
what other people are doing, but also those with whom we compare ourselves. 

AUcott (2011) 

Status quo bias 

The current state of the world dominates people’s decision making. It is hard 
to imagine that the world will be different tomorrow, or five minutes from now, 
and often people prefer an outcome simply because it is the status quo. 

Samuelson and 

Zeckhauser (1988) 

Trust 

The willingness to place one's resources at the disposal of another party out- 
side of a formal legal framework. 

Simpson (2007) 

Visual cue 

Visual features of an environment that can influence intuitive or unconscious 

decision making. 

Wansink, Painter, 
and North (2005) 
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APPENDIX TABLE A.2 

REASONS FOR NOT RENEWING AT ANY APPOINTMENT, ROUND 1 

INDIANA OFFICE OF EARLY CHILDHOOD AND OUT-OF-SCHOOL LEARNING 



PROGRAM 

CONTROL 


OUTCOME 

GROUP 

GROUP 

DIFFERENCE 

Reasons for not completing the redetermination 
process at any appointment (%) 

Missing or incomplete pay stubs 

11.0 

11.0 

-0.1 

Missing or incomplete other forms showing income 

7.3 

7.5 

-0.1 

Missing or incomplete school documents 

3.8 

4.2 

-0.4 

Problem with provider information sheet 

3.3 

3.9 

-0.6 

Missing or incomplete proof of address 

3.2 

3.8 

-0.7 

Missing proof of identity 

0.6 

0.4 

0.2 

Sample size (total = 5,332) 

2,666 

2,666 



SOURCE: MDRC calculations based on data from the Office of Early Childhood and Out-of-School Learning. 

NOTES: Estimates were regression-adjusted using ordinary least squares, controlling for pre-random assign- 
ment characteristics of sample members. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 
Rounding may cause slight discrepancies in differences. 


APPENDIX TABLE A.3 

REASONS FOR NOT RENEWING AT ANY APPOINTMENT, ROUND 2 

INDIANA OFFICE OF EARLY CHILDHOOD AND OUT-OF-SCHOOL LEARNING 


OUTCOME 

PROGRAM 

GROUP 

CONTROL 

GROUP 

DIFFERENCE 

Reasons for not completing the redetermination 
process at any appointment (%) 

Missing or incomplete pay stubs 

17.2 

15.8 

1.4 

Missing or incomplete other forms showing income 

2.8 

2.4 

0.5 

Missing or incomplete school documents 

2.8 

2.4 

0.5 

Problem with provider information sheet 

4.8 

4.2 

0.6 

Missing or incomplete proof of address 

3.7 

3.2 

0.5 

Missing proof of identity 

0.3 

0.4 

0.0 

Sample size (total = 4,732) 

2,365 

2,367 



SOURCE: MDRC calculations based on data from the Office of Early Childhood and Out-of-School Learning. 

NOTES: Estimates were regression-adjusted using ordinary least squares, controlling for pre-random assign- 
ment characteristics of sample members. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 
Rounding may cause slight discrepancies in differences. 
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APPENDIX FIGURE A.l 
RESEARCH DESIGN AND SAMPLES 

INDIANA OFFICE OF EARLY CHILDHOOD AND OUT-OF-SCHOOL LEARNING 


Families apply for CCDF 
subsidy and are placed on 
the waitlist. 



PTQ intervention 

(June-October 2014) 

Total sample (children) 
N = 12,652 


Cohort 1 
N = 5,949 

Random assignment 
date: 

6/12/14 


Cohort 2 
N = 6,703 

Random assignment 
dates: 

6/19/14-9/29/14 


Control 
N = 4,255 


Referrals-only 
N = 1,191 

Referrals-phone 
N = 503 


Control 
N = 2,402 


3 

3 

3 


Referrals-only 
N = 2,389 

Referrals-phone 
N = 1,912 


3 

3 

3 


Redetermination, 
Round 1 

j (January-June 2014) 

Total sample 
(families) 

N = 5,332 


K Control 

N = 2,666 J 

l ^ 

K Program | 
N = 2,666 J 


/ "" \ 
Redetermination, I 
Round 2 

(July-November 
2014) 


Total sample 
(families) 

N = 4,732 


K Control 

N = 2,367 J 

I 

K Program | 
N = 2,365 J 
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APPENDIX FIGURE A.2 

HIGHLY RATED PROVIDER SELECTION OVER TIME IN COHORT 2 
BY RESEARCH GROUP, PTQ TEST 

INDIANA OFFICE OF EARLY CHILDHOOD AND OUT-OF-SCHOOL LEARNING 



Days on waitlist before enrollment 


SOURCE: MDRC calculations based on data from the Office of Early Childhood and Out-of-School Learning. 
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APPENDIX B 

Technical 

Appendix 
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This appendix addresses the specific clustering, weighting, and exploratory analysis methods used to analyze 
the Paths to QUALITY (PTQ) outcomes. 


Estimation Equation 

The primary analysis is an intent-to-treat analysis, which estimates differences between randomized groups 
whether or not everyone ultimately received the intended treatment for that group. The child-level impacts are 
estimated by regression of the outcome on treatment assignment using a linear model with standard errors 
clustered by family. Baseline characteristics are included as covariates to increase the precision of estimated 
impacts. In notation, the impacts are calculated using the following statistical model: 

Y. = a + BE.+ 8X..+ v.. 

ij I j ij ij 

Where: 

Y = outcome for child i in family j 
a = the mean outcome for control families 
= the effect of the program on outcome Y 
E. = indicator for assignment to the program group for family j 
S = a vector of coefficients for the baseline characteristic covariates 
X = a matrix of baseline characteristics for child i in family j: 

• number of times a parent has received a CCDF subsidy 

• number of children in the family 

• an indicator for having an infant 

v.. = a random error term comprised of u. + e.. 
y j y 

u . = a random error for family; 
e = a random error for child i within family j 


Weighting 

Because of the differences in random assignment ratios between Cohorts 1 and 2 discussed on page 13 of 
Chapter 2, inverse probability of treatment weights were applied to the adjusted impact estimates to account 
for differing random assignment ratios across cohorts: 1 



Where: 

A ip TW - inverse probability of treatment weighted outcome difference between program and control groups 


p.. = probability of assignment to the program group for child i given family ;"s randomization cohort 

Under this weighting scheme, each cohort’s contribution to the impact estimates is proportional to its sample 
size. There is no assumption made of constant treatment effects across cohorts. 2 The difference in waitlist 
times between the cohorts likely contributed to differing impacts. 


1 Lunceford and Davidian (2004). 

2 Hong and Raudenbush (2008). 
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Exploratory Analyses 

Appendix Table B.l shows the PTQ outcome averages among Cohort 2 members who enrolled in CCDF. As 
discussed in Chapter 2, a substantial proportion of families did not enroll in the CCDF program and therefore 
parents could not have been observed choosing a PTQ provider. A nonexperimental analysis is presented here 
showing the rates of PTQ provider selection for children in families that did enroll. Because this subsample is 
limited to families that chose to enroll after random assignment, it is possible that there are differences be- 
tween those enrolled in the program group and those enrolled in the control group, despite similar enrollment 
rates, which could contribute to any differences observed between the outcomes of the research groups in this 
table. For this reason, the table is presented for descriptive purposes only, without statistical tests. 


APPENDIX TABLE B.l 

PROVIDER SELECTION OUTCOMES AMONG THOSE WHO 
ENROLLED IN THE CCDF PROGRAM 
FOR COHORT 2, PTQ TEST 

INDIANA OFFICE OF EARLY CHILDHOOD AND OUT-OF-SCHOOL LEARNING 


OUTCOME 

CONTROL 

GROUP 

REFERRALS- 

ONLY 

GROUP DIFFERENCE 

REFERRALS- 

PHONE 

GROUP 

DIFFERENCE 

OVERALL 

AVERAGE 

Provider quality-rating selection 

Selected an unrated provider (%) 

30.6 

34.4 

3.8 

27.0 

-3.6 

30.9 

Selected a PTQ provider (%) 

69.4 

65.6 

-3.8 

73.0 

3.6 

69.1 

Level 3 or 4 provider 

29.6 

31.3 

1.8 

36.4 

6.8 

32.2 

Level 4 provider 

13.8 

17.4 

3.5 

18.8 

5.0 

16.6 

Level 3 provider 

15.8 

14.0 

-1.8 

17.7 

1.9 

15.7 

Level 2 provider 

7.6 

8.9 

1.3 

7.4 

-0.2 

8.0 

Level 1 provider 

32.2 

25.4 

-6.8 

29.2 

-3.0 

28.9 

Sample size (total = 2,997) 

1,041 

1,072 


884 




SOURCE: MDRC calculations based on data from the Office of Early Childhood and Out-of-School Learning. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

This table only shows nonexperimental comparisons. Significance tests are not calculated for nonexperimental data. 
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